Multifocal imaging systems and method

ABSTRACT

In the systems and methods of the present invention a multifocal multiphoton imaging system has a signal to noise ratio (SNR) that is reduced by over an order of magnitude at imaging depth equal to twice the mean free path scattering length of the specimen. An MMM system based on an area detector such as a multianode photomultiplier tube (MAPMT) that is optimized for high-speed tissue imaging. The specimen is raster-scanned with an array of excitation light beams. The emission photons from the array of excitation foci are collected simultaneously by a MAPMT and the signals from each anode are detected using high sensitivity, low noise single photon counting circuits. An image is formed by the temporal encoding of the integrated signal with a raster scanning pattern. A deconvolution procedure taking account of the spatial distribution and the raster temporal encoding of collected photons can be used to improve decay coefficient. We demonstrate MAPMT-based MMM can provide significantly better contrast than CCD-based existing systems.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/442,702, filed May 25, 2006, which claims the priority of U.S.Provisional Application No. 60/684,608 filed May 25, 2005 entitled,MULTI FOCAL MULTIPHOTON IMAGING SYSTEMS AND METHODS, the whole of eachof these applications being hereby incorporated by reference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under grant numbers R33CA84740 and R33 CA91354 awarded by the National Institutes of Health.The government has certain rights in this invention.

BACKGROUND OF THE INVENTION

Systems and methods for microscopic analysis of biological material havebeen used for characterization and diagnosis in many applications.Fluorescence microscopy, for example, has been used for optical analysisincluding the histological analysis of excised tissue specimens. Opticalcoherence tomography has been used for three dimensional imaging oftissue structures, however, the limited resolution of existing systemshas constrained its use for definitive pathological analysis. Confocalmicroscopy has been used for high resolution imaging and hascontrollable depth of field but limited imaging speed.

Multiphoton microscopy is based on the nonlinear excitation offluorophores in which fluorescence generation is localized at the focusof excitation light. Multiphoton microscopy is used for deep tissueimaging because of its subcellular three dimensional (3D) resolution,minimal phototoxicity, and tissue penetration depth of over a fewhundred micrometers. It has become useful in biomedical studies such asneuronal plasticity, angiogenesis in solid tumors, transdermal drugdelivery, and non-invasive optical biopsy, for example.

A practical limitation of multiphoton microscopy is its imaging speedwhich typically lies in a range of less than two frames per second.While this speed is sufficient in many cases, there remain applicationsin which can be enhanced by improvements in imaging speed. There is acontinuing need for further improvements in microscopic analysis ofbiological materials for numerous applications.

SUMMARY OF THE INVENTION

The present invention relates to systems and methods for the multifocalimaging of biological materials. An optical system is provided in whicha plurality of optical pathways are used in combination with focusingoptics to provide a plurality of focal locations within a region ofinterest of a material being optically measured or imaged. The detectorcan comprise a plurality of detector elements which are correlated withthe plurality of focal locations to provide for the efficient collectionof light from the material being imaged. A preferred embodiment of theinvention utilizes a scanning system that provides relative movementbetween the material and the focal locations to provide for fast imagingof the material.

In a preferred embodiment a light source, such as a laser, is used witha multifocal optical element to provide an array of spatially separatedoptical pathways. The multifocal optical element can comprise a microlens array, a diffractive optical element, or a beam splitter device,for example, such that a plurality of beams are provided that can befocused onto a plurality of focal locations within a biological materialto be imaged.

An important issue in the collection of light from discrete focal spotsor locations within a turbid medium such as tissue is the cross talkthat can occur due to the scattering of light. This cross talk cansubstantially limit the usefulness of the images of the tissue that areproduced. By increasing the distance between adjacent focal spots suchcross talk can be reduced or eliminated, however, this reduces theresolution of the resulting image or increases the time needed to scanthe tissue. Thus it is desirable to employ focal spacing of at least 10microns and preferably more than 25 microns.

In a preferred embodiment of the invention, high speed multiphotonmicroscopy can measure biological systems such as, for example, kineticprocesses in the cytosol of a single cell, for example, or imaging avolume of tissue. For example, high speed 3D imaging can map 3Dpropagation of a calcium wave and the associated physical contractionwave through a myocyte, or the rolling of luckocytes within the bloodvessel of a solid tumor. High speed 3D microscopy provides for samplinga statistically significant volume of biological specimens. Since thefield of view of most microscopes is limited to about 100 microns on aside with an imaging depth of 100 microns, the measurement volume islimited to only 1×10⁻³ mm³. While this volume is sufficient for cellularimaging, many tissues have physiologically relevant structures rangingfrom the cellular level up to several millimeters in size. For example,a neuron with its extensive dendritic tree can span a volume over 1 mm³and many dermal structures such as hair follicles and sabestious glandscan not be seen with images confined to an area of 100-200 micrometers.It is desirable, for example, to image a hierarchy of cardiac structuresranging from a single nucleus in a cardiac myocyte, to the distributionof muscle fibers and blood vessels, to the structure of chambers andheart valves with perfect registration across five orders of magnitudeby imaging a whole mouse heart. Equally importantly, traditional 3Dmicroscopes sample only tens to hundreds of cells and can never achievecomparable statistical accuracy and precision in many biomedical assaysas techniques such as flow cytometry and image cytometry. High speedimaging can circumvent this difficulty by improving the number of cellsor tissue volume to be sampled. By performing high speed multiphotonimaging, better quantitative measurements of transport pathways acrossthe stratum corneum in transdermal drug delivery applications can bemade, for example.

Systems and methods have been developed to enhance multiphoton imagingspeed. A first method increases the scanning speed by using a high-speedscanner such as a polygonal mirror scanner or a resonant mirror scannerinstead of a galvanometer-driven mirror scanner. This achieves anincrease of scanning speed of more than 10 frames per second in theimaging of typical tissue specimens. In general, the system can operateat frequencies in a range of 1 to 500 Hz. This method can be used forturbid tissue imaging since it is not sensitive to the scattering ofemission photons. A second method increases the imaging speed byparallelizing the multiphoton imaging process. It scans a sample with amultiple of excitation foci instead of forming only a single focus.These foci are raster scanned across the specimen in parallel where eachfocus needs to cover a smaller area. The emission photons from thesefoci are collected simultaneously with a spatially resolved detector.One advantage of this method is that the imaging speed is increased bythe number of excitation foci generated, without increasing the power ofexcitation light per each focus. High speed scanning systems needshigher power to compensate for the signal reduction per pixel due to thedecrease of pixel dwell time. Images can be obtained by selecting thedepth of focus to be positioned in a plane within the tissue or sampleat a depth in a range of 10 microns to 500 microns.

In another embodiment, fiber optics can be used to couple the lightsource to the microlens array or other beam splitting element. Thesystem can be implemented as a handheld optical probe for the diagnosisof dermal, cervical or colorectal cancer, for example.

The brain is an inherently three dimensional organ composed of manysubregions. Accurate segmentation of brain morphology of small mammalsis currently challenged by the lack of techniques which can sample thebrain at high resolution over a large volume. The current method ofchoice, serial section reconstruction, is laborious, time consuming, anderror prone. The device and methods described herein can quickly imagebrains or thick tissue sections of brains in 3D at sufficient resolutionand over a large enough volume to provide 3D images suitable forclassification of brain morphology and biochemical composition. Thebrain can be further stained by dyes, such as nuclear dyes DAPI orHoescht, either through intravital injection, transgenic expression, orex vivo methods, to facilitate classification of regions. Automaticsegmentation routines can also be used to improve the classification andautomate portions of the process.

Accurate measurement of vasculature is important to characterize manybiomedical for vasculature related diseases. For instance,proangiogenesis therapies are useful in such areas as tissueengineering, wound healing, bone fractures and coronary heart disease.Anti-angiongenesis treatments are important in processes as cancer,blindness, and rheumatoid arthritis. Unfortunately traditionalhistopathological analysis of tissue sections is wholly inadequate tocharacterize the vasculature of a tissue or organ as blood vessels formcomplex, multiscale 3D networks, with feature spanning from thesubmicron to centimeter scale. The device and methods described in thepatent are capable of acquiring high quality 3D datasets over 3D tissueand organ samples suitable for characterization of the vasculature ofthe tissue. To aid visualization of the vasculature, the tissue can bestained by contrast agents which bind to the epithelial wall of theblood vessels, or fill the interior of vessels. Automatic segmentationroutines can also be used to improve the classification and automateportions of the process.

A large percentage of deaths are due to metastasis. Unfortunately, themigration of cancer cells from the primary tumor to secondary sites is amulti-step process which is not well understood. Standardhistopathological analysis is ill-suited to study metastasis and suffersfrom a number of limitations. First, it is extremely difficult to findrare metastatic cancer within a 3D bulk tissue using traditional 2Dhistopathology. In many instances traditional 2D histopathology isunable to find evidence of the presence of metastatic cancer cells in anorgan of animal. However, it is known that many subjects eventuallydevelop tumors at a later time. It is clear that traditionalhistopathology cannot effectively detect rare cells. Another limitationis that the present histopathology methods provide limited informationabout the 3D spatial arrangement of cancer cells with the 3D vasculatureof the organ. It is known that one of the critical steps in metastasisis extravasation into the surrounding stroma from the vasculature so itis essential to be able to visualize this spatial relationship betweencancer cell and the endothelial blood vessel wall. Preferred embodimentsof the present invention are capable of acquiring high quality 3Ddatasets over 3D tissue and organ samples suitable for characterizationof the metastases. To aid visualization of the metastases, the cancercell can be stained by dyes or labeled with proteins such as OFP.Automatic segmentation routines can also be used to improve theclassification and automate the localization of the cancer cells andtumors.

In order to understand the effects of a drug on an organism, analysis atthe tissue, whole organ, and whole organism level is vitally important.ADME, efficacy and toxicology effects are known to have strong spatialvariations on the morphological, cellular and biochemical state of atissue. Even within a specific tissue type, the response can benonuniform due to variations in the transport and distribution of a drugthroughout tissue, epigenetic expression, and cellular activity. Thedevices and methods described herein can be used to providemorphological, biochemical and spectroscopic information about the stateof a tissue across multiple length scales, from subcellular, wholetissue, whole organ and even entire organism, in response to thetreatment of a molecular agent. Efficacy, ADME, and toxicologyinformation can be derived which provides a fuller and more accuratedescription to predict the actual effect of drug candidate at theorganism level.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an imaging system in accordance with apreferred embodiment of the invention.

FIGS. 2a-2c are images of human skin acquired with the present inventionincluding the stratum corneum layer, stratum granular and the basallayer, respectively.

FIG. 3 graphically illustrates the signal decay with increasing imagingdepth of conventional systems and those incorporating the presentinvention.

FIGS. 4a-4d include images before and after deconvolution as well asgraphical illustration of scattering and crosstalk.

FIG. 5 illustrates images based on CCD, MAPMT and deconvolution thereofat the surface and at different depths of brain tissue.

FIG. 6 illustrates a method and apparatus for multi-focal, multi-photonmicroscopy (MMM) according to a preferred embodiment of the invention,showing parallelized illumination and detection device with a commonfocusing device.

FIG. 7 illustrates a method and apparatus for multi-focal, multi-photonmicroscopy (MMM) employing scanning and multi anode PMT's, according toa preferred embodiment of the invention.

FIG. 8 illustrates another method and apparatus for multi-focal,multi-photon microscopy (MMM) employing scanning and multi anode PMT' s,according to a preferred embodiment of the invention.

FIG. 9 illustrates generating and detecting a 3D foci pattern in a focalregion.

FIGS. 10(a)-(i) illustrates close up views of the 3D focal regiongenerated by the setup in fig. W-10 and views of the 3D scanning: (a)the focal region; (b) an array of excitation light beams; (c) x/y viewat first depth; (d) x/y view at second depth; (e) x/z view; (f) multiplerows of excitation foci lie in different focal planes and are all shownin this xy view; (g) xz view of the rows shown in (f); (h) x/y view ofthe x/y scanning configuration covering the x-y-z image (as in (f) allfoci are shown, even though they lie in different planes and in thelower part of the images even behind each other; and (i) a view in theyz plane illustrating scan progression.

FIG. 11a illustrates a further method and apparatus for a 3D cytometer,based on multi-focal, multi-photon microscopy (MMM) employing scanningand multi anode PMT's, according to a preferred embodiment of theinvention.

FIG. 11b illustrates an image of the array of foci in the focus of theobjective lens; the foci are 45 micrometers apart resulting in ascanning field of 240 mm when 6×6 foci are utilized.

FIG. 11(c) illustrates the Z-profile and corresponding fit 25 functionof a 200 nm bead.

FIGS. 12(a)-(e) illustrates a further method and apparatus for multicolor detection MMM employing scanning and multi anode PMT's accordingto a preferred embodiment of the invention: (a) the setup in thexz-plane; (b) the foci and their scanning in the focal xy-plane; (c)detection path in the yz-plane (d) detection path projected in thexz-plane; (e) the anodes of the multi anode PMT in the x/z and x/y planein conjunction with the detected colors. In this case only visible lightis shown. Any other light spectra can be separated, though.

FIG. 13 illustrates a beam splitter configuration used in someembodiments according to the invention.

FIGS. 14(a)-(d) illustrate preferred embodiments for providingillumination beam paths in accordance with the invention.

FIGS. 15(a)-(d) illustrated further preferred embodiments for detectinglight from different focal locations in accordance with preferredembodiments of the invention.

FIG. 16 illustrates determining the optimal number of foci at a certainlaser power for samples with different damage thresholds.

FIGS. 17(a) and (b) illustrate a time multiplexing method.

FIGS. 18(a)-(c) illustrate a pixellated detector collection method.

FIG. 19 illustrates an endoscope apparatus according to an embodiment ofthe invention.

FIGS. 20(a) and (b) illustrates scattered light detection with one PMTand one excitation focus according to an embodiment of the invention.

FIGS. 21(a) and (b) illustrates scattered light detection with two PMTsand one excitation foci according to an embodiment of the invention.

FIGS. 22(a) and (d) illustrates scattered light detection with two PMT'sand two excitation foci according to an embodiment of the invention.

FIGS. 23(a) and (b) illustrates reducing optical cross talk byincreasing the distances between the excitation foci and distancesbetween the detection elements.

FIGS. 24(a) and (b) illustrates reducing optical cross talk according toan embodiment of the invention by increasing the distance between theexcitation foci and increasing the area of the detection elements.

FIGS. 25(a)-(e) illustrate in tabular form two alternative embodiments Aand B of the invention in terms of changing optical setup.

FIGS. 26(a)-(b) illustrates the different conjugated areas of detectionfrom each channel of the multi anode PMT in the conjugated image planefor configuration A and B from FIG. 22(a)-(e).

FIGS. 26(c)-(d) illustrate an objective lens with large field of viewenables large separation of foci and thus enables low optical crossreduction

FIGS. 27(a) and (b) illustrate data post-processing sequences.

FIGS. 28(a) and (b) illustrate a normalization method.

FIGS. 29(a)-(c) illustrate a linear deconvolution process.

FIGS. 30(a)-(d) illustrate further details for a linear deconvolutionprocess I: Signal distribution in multi channel detector

DETAILED DESCRIPTION OF THE INVENTION

As the input power of excitation light increases, the signal isincreased proportionally to the square of input power, S(t)∝[P(t)]².However, there is a limitation in input power level due to finitelifetimes of fluorophores.

In the multiphoton excitation of fluorophores with a pulsed laser, thefluorophores, which are excited with the last pulse, stay in theexcitation state for a few nano-seconds (depending on the fluorophore).Some excited fluorophores may not be excited again with the next pulseof excitation light (12 ns later in case of the laser having 80 MHzpulse repetition rate). Therefore, signal level becomes saturated with ahigher input power than the limited input power level. The limitation onthe input power level is related to the excitation probability of asingle fluorophore with a single pulse, P_(pulse). It is formulated inthe following expression with the condition that excitation light isfocused with an objective into a fluorophore of an absorptioncoefficient (δ_(a)).

$\begin{matrix}{P_{pulse} = {{\delta_{a}\left\lbrack \frac{\lambda}{K_{a}{hc}} \right\rbrack}^{2}{\frac{{NA}^{4}}{\tau_{p}f_{p}^{2}}\left\lbrack {P_{o}(t)} \right\rbrack}^{2}}} & (1)\end{matrix}$The nominal conditions are that the excitation light has the wavelength(λ=800 nm), the pulse width (τ_(p) =200 fs), the pulse repetition rate(f_(p)=80 MHz), and the average power, P_(o)(t). The numerical aperture(NA) of the lens objective is 1, (NA=1). The fluorophore has theabsorption coefficient, (δ_(a)=10 GM), where 1 GM is 10⁻⁵⁰ cm⁴×s/photon.In order to avoid the saturation, P_(pulse) must be less than 0.1 ingeneral (P_(pulse)<0.1). With these conditions, the input power(P_(o)(t), with which P_(pulse)becomes close to the saturation limit, isapproximately 6 mW, (P_(o) ^(sat)(t)≅6 mW) in this example. In case theconcentration of the fluorophores is 10 μM, the number of emissionphotons collected per second is approximately 3×10⁷ photons/s with theassumption that the collection efficiency of emission photons isapproximately 0.01 (ε_(col)=0.01). Assuming that each pixel needs 300photons and each image comprises of 256×256 pixels, the frame rate thatcan be achieved with the input power under the saturation limit is 1.5frames/s. Although the higher frame rate is achieved with specimens ofhigher fluorophore concentration, it is clear that there is a limitationin input power level due to fluorophore saturation.

MMM increases the frame rate by scanning with multiple excitation foci.Therefore, MMM can achieve the higher frame rate, while the input powerfor each excitation focus is kept below the saturation limit. Forexample, the MMM system, which scans with an 8×8 array of excitationfoci, can achieve the frame rate of 96 frames/s (=1.5 frames/s×64 foci).In a preferred embodiment it is desirable to collect at least 15 framesper second and preferably 30 frames per second or more. One practicallimitation in MMM is that more input power is required to generatemultiple excitation foci. The power requirement to generate 64 foci is384 mW (=64 foci×6 mW per each focus). Since available laser sources canoutput approximately 2 W of power, enough power is available for MMM.

The limit of optical imaging depth in tissues is limited by photoninteraction with tissue constituents. Photon scattering is a dominantfactor in multiphoton microscopy whereas the effect of photon absorptionis relatively negligible. Scattering of excitation photons reduces theamount of fluorescence generated at its focus, because less excitationphotons reach the focal volume. The emission photons from the focus arealso scattered so that they may not be collected by the optics in thedetection path or spatially dispersed in the imaging plane wheredetectors are positioned. Since the excitation light has a longerwavelength than the emission light, the excitation light typicallyexperiences less scattering than emission light. The effect of photonscattering is expressed by the mean free path length of scattering, Pwhich is the depth constant in exponential decay of unscattered photons,S(z)∝ exp(−z/l^(s)).

Intralipid emulsion can be used as a tissue phantom with similar opticalproperties as tissue. The optical properties of 2% intralipid are meanfree path length at excitation wavelength (780 nm) of 167 μm, (l_(ex)^(s)≅167 μm) and at emission wavelength (515 nm) of 62.5 μm, (l_(em)^(s)≅65 μm). Since it is known that only ballistic excitation photonscontribute multiphoton excitation in the depth of a few times ofscattering length, the amount of multiphoton excitation decays with themean free path length of 84 μm (=167 μm/2) with the consideration thattwo-photon excitation is a quadratic process. Conventional multiphotonmicroscopy is based on the scanning of a single excitation focus and thesignal is collected using a detector without spatial resolution such asa conventional photomultiplier tube (PMT). The PMT has a large detectionarea and can collect most of the signal generated at the excitationfocus including a large fraction of the scattered photons. Therefore,conventional multiphoton microscopy is relatively immune to thescattering of emission photons by the tissue. However, for an MMM systemthat utilizes a CCD detector to distinguish the signals originated fromeach of the foci, the scattering of emission photons seriously degradesthe SNR of the instrument for deep tissue imaging. The CCD camera hasrelatively slow readout speed and typically integrates all the emissionphotons during the acquisition of each frame. Because a CCD cameracontains pixels in which each pixel covers a 0.1 μm² region in thespecimen plane, scattered emission photons deflected from their originalpaths are not collected in the correct pixel but are distributed broadlyacross the imaging plane. The distribution of scattered emission photonsis very broad with its FWHM of 40 μm in the depth of 233 l_(em) ^(s).These scattered photons result in a degradation of image SNR by morethan one order of magnitude when imaging depth is over 2×l_(em) ^(s),compared with conventional multiphoton microscopy.

The major limitation of CCD-based MMM system lies in its small pixelarea. For conventional wide field imaging, a large number of CCD pixelsare needed to maintain good resolution while covering a good size fieldof view. A 100 □m size image will require about 10⁷ pixels to be imagedat full optical resolution (300 nm). The situation is very different forMMM imaging. Since a femtosecond light source can only provide atmaximum 2-4 watts of optical power and typically about 50-100 mW arerequired at each focus to generate an efficient muliphoton excitationprocess for deep tissue imaging. An MMM system can realistically andeffectively scan about 20-40 foci in parallel with tissue specimens.Since these foci are raster scanned across the specimen, the imageresolution is determined by the excitation point spread function (PSF)of the light and is not sensitive to the detector pixelation. Inparticular, a preferred embodiment uses an MMM system having photondetectors containing only as many elements as the number of excitationfoci. The need for fewer elements allows the use of a detector with asignificantly larger pixel area while maintaining a reasonable devicesize. A multi-anode PMT (MAPMT) is a preferred detector for thispurpose.

A preferred embodiment of the present invention uses an MAPMT instead ofthe CCD camera for the signal collection from multiple foci. The MAPMTis similar to conventional PMTs with a good quantum efficiency (over 20%in the blue/green spectral range), negligible read noise and minimaldark noise with cooling. MAPMT has a cathode and dynode chain with ageometry that ensures that the spatial distribution of photons on thecathode is reproduced accurately as electrons distribution at the anode.The anode of the multi-anode PMT is divided rectilinearly into itselements providing spatial resolution for the simultaneous collection ofsignals from multiple locations. In one example, a MAPMT, which has anarray of 8×8 pixels (H7546, Hamamatsu, Bridgewater, N.J.) is used. Notethat a flat panel detector having a pixel area of sufficient size canalso be used. For example, a binnable CMOS or CCD imaging sensor can beoperated to read out binned images at comparable frame rates with aneffective pixel size corresponding to that of a MAPMT.

A preferred embodiment of the invention uses the imaging systems asdescribed herein in conjunction with a system for sectioning a samplesuch as a tissue sample that is described in greater detail in U.S.patent application Ser. No. 10/642,447, by So, et al. filed Aug. 15,2003, the entire contents of which is incorporated herein by reference.

The schematic of a preferred embodiment of the imaging system 10 inaccordance with the invention is shown in FIG. 1. The light source 12used is a Ti-Sapphire (Ti-Sa) laser (Tsunami, Spectra-Physics, MountainView, Calif.) pumped by a continuous wave, diode-pumped,frequency-doubled Nd:YVO₄ laser (Millenia, Spectra-Physics, MountainView, Calif.). It generates approximately 2 W at 800 nm wavelength whichis sufficient for most MMM applications. The excitation beam from thelaser is optically coupled using optical fiber 14 or free space lenssystem to a beam expander 16 and then illuminates a microlens array 20(1000-17-S-A, Adaptive Optics, Cambridge, Mass.) which, in this example,is an array of 12×12 (or 8×8) square microlenses that are 1 mm×1 mm insize and 17 mm in focal length. The degree of beam expansion can beselected such that an array of 8×8 beam-lets is produced after themicrolens array. The beam-lets are collimated after lens L1 andreflected onto an x-y scanner mirror 30 (6220, Cambridge Technology,Cambridge Mass.) which is positioned in the focal plane of lens L1. Inthis configuration, the beam-lets overlap each other on the scannermirror surface and are reflected similarly by the rotation of thescanner mirror. After the scanner, the beam-lets enter a coupling lenssystem such as a microscope (BX51, Olympus, Melville, N.Y.) via amodified side port. A combination of lenses L2 and L3 expands thebeam-lets to fill the back aperture of the objective lens 36 in order touse the full NA of the objective lens. The scanning mirror is in thetelecentric plane of the back aperture of an objective lens so that thebeamlets are stationary on its back aperture independent of the motionof the scanner mirror. The objective lens generates the 8×8 focus arrayof excitation light in the sample plane in the specimen 34. The scannermirror moves the array of excitation foci in the sample plane in araster pattern to cover the whole sample plane. Alternatively, a digitalmicromirror (MEMS) device can be used to control beam scanning in thesample plane. A beamsplitter can also be used to split an input beambefore the microlens array. Another alternative embodiment employs adiffractive optical element in conjunction with a beam splitter. Theobjective used in this system is a 20×water immersion lens with 0.95 NA(XLUMPLFL20XW, Olympus, Melville, N.Y.). The excitation foci areseparated from each other by 45 μm in this example so that the scanningarea of each focus is 45 μm×45 μm. The frame size is 360 μm×360 μm byscanning with the array of 8×8 foci. The frame rate to generate imagesof 320×320 pixels becomes approximately 19 frames per second with thepixel dwell time of 33 μs.

Emission photons are generated at the array of excitation foci in thespecimen and are collected by the same objective lens forming an arrayof emission beam-lets. In case of a CCD-based MMM, the emissionbeam-lets are reflected on a long-pass dichroic mirror 38 (650dcxxr,Chroma Technology, Brattleboro, Vt.) and get focused in optional CCDcamera 28 (PentaMax, Princeton Instruments, Trenton, N.J.) with a lens(L3). The CCD camera integrates emission photons during the scanningtime of each frame to generate images. In case of a preferred embodimentusing an (without the CCD) MAPMT, the emission beam-lets travel back tothe scanner mirror 30 retracing the excitation paths. The emissionbeam-lets are reflected by the scanner mirror. The emission beam-letsare de-scanned and their propagation directions remain stationaryirrespective of the movement of the scanner. The emission beam-lets arereflected by a long-pass dichroic mirror 32 (650dcxxr, ChromaTechnology, Brattleboro, Vt.) and are focused after lens (L4). Ashort-pass filter (E700SP, Chroma Technology, Brattleboro, Vt.) blocksany strayed excitation light. The focused emission beam-lets arecollected at the center of corresponding channels of a MAPMT 22 (H7546,Hamamatsu, Bridgewater, N.J.). The emission photons coming from thearray of excitation foci are collected by the MAPMT. An image is formedby the temporal encoding of the integrated signal with the known rasterscanning pattern using image processor or computer 24 and iselectronically stored in memory and/or displayed using display 26.

The pair of L2 and L4 lenses magnifies the array of emission foci sothat individual emission beamlets are focused at the center ofcorresponding elements of the MAPMT. Further, since the emissionbeam-lets are descanned, they remain stationary. Since the emissionbeam-lets have to go through more optical elements, loss of emissionphotons occurs. The transmission efficiency is approximately 0.7. Thesignals from the MAPMT are collected by a stack of four multi-channelphoton counter card (mCPhC) which has 64 channels for simultaneoussignal collection. Each mCPhC has 18 channels of photon counter circuitsand can be housed 25 with a digital interface to the computer 24. ThemCPhC expandable so that 64 channels are readily implemented by using 4cards in parallel. The mCPhC has a 32-bit parallel interface with acomputer for high-speed data transfer. Currently, the speed is limitedby the speed of the computer PCI bus. Transfer rate can be more than onehundred frames (320×320 pixels, 16 bit images) per second.

Since the scattered emission photons have the spatial distribution of 40μm as its FWHM at the imaging depth of 2×l_(em) ^(s), the sensitivity ofthe microscope is partly determined by the effective detector area, thearea in the sample plane from which a detector collects emissionphotons. Since microscopes are telecentric systems, the effectivedetector area is linearly related with the detector size in the imageplane. With a magnification, M, and a linear dimension of detector,L_(D), the linear dimension of effective detector area (L_(E)) isL_(E)=L_(D)/M. In general, the larger the effective detector area, themore effective the detector can collect scattered emission photons. Inthe case of using a 20 × magnification objective, a 10 mm diameterstandard PMT has an effective detector area of 500 μm diameter that issignificantly larger than the width of the scattered emission photondistribution. Therefore, standard PMTs have good collection efficiencyof scattered emission photons and allow very effective deep tissueimaging. In the case of a spatially resolved detector, each pixel can betreated as an individual detection element. For a CCD camera with 20μm×20 μm pixels, each pixel has an effective detector area of 1 μm×1 μmfor 20 × magnification. Therefore, the CCD-based MMM system cannotutilize these scattered emission photons which are distributed uniformlythroughout the image contributing to the background noise. In thisexample of the MAPMT-based MMM system, the effective detector area ofeach channel is 45 μm×45 μm. Therefore, the MAPMT can collectsignificantly more scattered emission photons into the correct channelsthan the CCD camera, because its effective detector area, or detectorelement collection area, is comparable with, or corresponds to, thewidth of the scattered photon distribution from each focal area (45microns×45 microns).

MAPMT-based MMM system can be easily converted to the conventionalmultiphoton microscope which is based on single-focus scanning andsignal collection with PMTs. In the set up of conventional multiphotonmicroscope, the excitation beam is not expanded and goes directly ontothe scanner without the combination of the microlens array and lens Li.The rest of excitation beam path is the same as MAPMT-based MMM.Specimens are scanned with a single excitation focus. The emission lightcollected by the objective lens is reflected on a dichroic mirror. Thereflected emission beam shrinks with a pair of lenses and is collectedby a detector (PMT). An image is formed by the temporal encoding of theintegrated signal with the known raster scanning pattern.

CCD-based MMM have limitations for turbid tissue imaging by measuringthe effect of emission photon scattering on PSF (scattering function).Scattered emission photons form additional intensity distribution aroundthe PSF constructed with ballistic unscattered photons. Their intensitydistribution is quite wide with its FWHM of 40 μm at the imaging depthof 2×l_(em) ^(s). FWHM of the total PSF (including the intensitydistribution due to scattered emission photons) is not changed due toscattering up to such depth because the wide distribution of scatteredemission photons does not contribute to FWHM. In terms of contrast,signal decay in CCD-based MMM with the increase of imaging depth ishigher than that of SMM by an order of magnitude at 2×l_(em) ^(s). Alsothe wide distribution of scattered emission photons contributes as noiseand causes loss of contrast by another order of magnitude at the depth.

Imaging dermal structure based on autofluorescence has been performedusing the system of the present invention. Endogenous fluorophores havelow quantum yield and low extinction coefficients compared with typicalexogenous fluorescent labels. The dermal structure imaged using apreferred embodiment of the present invention has a layered structurewith significantly different indices of refraction resulting insignificant spherical aberration. Multiphoton imaging of dermalstructures without photodamage has a pixel rate of 15 KHz with 15 mWinput power. In this example an input power of 7 mW per focus at thespecimen with the excitation wavelength set at 800 nm. The objectiveused is 20×water immersion with 0.95 NA (XLUMPLFL20XW, Olympus,Melville, N.Y.). With a frame rate of 2.5 fps for a 320×320 pixel image(4 KHz pixel rate) this is 10 times faster than the previous systems.The epidermis is imaged down to the basal cell layer using thisMAPMT-based MMM. Representive layers from the stratum corneum, stratumgranular, and the basal layer are shown in FIGS. 2a-2c . The signal fromthese layers are mostly due to the fluorescence of NAD(P)H inside thecell. Thus an MAPMT-based MMM has equivalent or improved sensitivity asconventional multiphoton microscopy but with significantly increasedimaging speed. The intensity of the image is not uniform: the intensityis high in the center and becomes dim in the corner of the image. Thisis because the intensity of the excitation beam has Gaussian spatialdistribution so that the beamlets made from center part of the expandedbeam have higher intensity than the peripheral portions of the beam. Abeam splitter, serial dichroic mirrors or a top hat holographic filtercan be used to provide a more uniform array of beams delivered to theindividual focal positions.

Using both CCD and MAPMT detectors in MMM geometry, the signal decay canbe measured as a function of scattering length. As the imaging depthincreases, the signal is decreased due to scattering of both excitationphotons and emission photons. The signal decay is measured by imaging 4μm diameter fluorescent latex microspheres (F8858, Molecular Probes,Eugene, Oreg.) immobilized in 3D by 2% argarose gel (UltraPure LowMelting Point Argarose, Invitrogen, Carlsbad, Calif.). Intralipidemulsion (Liposyn III, Abbott Laboratories, North Chicago, Ill.) isadded to the sample as a scatterer in various concentrations of 0.5 to2%. Intralipid emulsion of 2% volume concentration is known to havesimilar scattering properties to those of tissues: mean free path length(l^(s)) of scattering is 80 μm, 168 μm at the wavelength of emission(605 nm), excitation (800 nm) respectively. The scattering properties ofthese intralipid solutions are verified by diffusive wave measurements.Peak intensity of the sphere image is a signal in the measurement andthe decay of peak intensity as a function of the imaging depth ismeasured at each concentration. The signal decay can also be measuredwith a conventional multiphoton microscope as a reference. Signal decaysin the three systems are measured down to a depth of 180 μm which isequivalent to 2.25×l_(em) ^(s) (FIG. 5). The signal decay is expressedan exponential function, S(z)=exp(−cz/l_(em) ^(s)). The decaycoefficient, c is 1.22, 1.87, 2.30 in case of the conventionalmultiphoton microscopy, MAPMT-based MMM, and CCD-based MMM respectively.The decay rate from the conventional multiphoton microscope is thelowest as expected. The decay is the combinational effect of bothexcitation and emission photon scattering. Since the effect ofexcitation photon scattering is the same, the difference in decaycoefficient is due to the effect of emission photon scattering. Thedecay coefficient, c from MAPMT-based MMM (1.87) is lower than the onefrom CCD-based MMM (2.30). However, the one from MAPMT-based MMM isstill higher than the one from the conventional multiphoton microscope.It indicates that the spatial distribution of scattered emission photonsis wider than the effective detector area of the MAPMT (45 μm×45 μm) sothat some portion of the scattered emission photons are collected in theneighboring channels. The ratio of intensity sum collected in theneighboring pixels of the MAPMT to the intensity in the correct pixelwas approximately 2 at the depth of 2×l_(em) ^(s).

Although a significant portion of the scattered emission photons arestill distributed outside the correct pixels in MAPMT-based MMM, thesephotons can be effectively restored to the correct pixels based onpost-acquisition image processing. Note that the photons acquired ateach pixel are temporally encoded and are organized to form an imagebased on the known scanner position as a function of time. This isexactly how images are formed in a conventional multiphoton or confocalmicroscope. A primary image is formed by photons acquired at the correctpixels corresponding to the fluorophore distribution in that portion ofspecimen. Note that the scattered photons in the neighboring pixels arealso similarly temporally encoded. Therefore, secondary “ghost” imagesare formed in the areas of the image covered by the neighboring pixels.As an example, FIG. 4a is an image of spheres at 150 μm deep from thesurface in 2% intralipid emulsion. The fact that the primary image atone pixel is “copied” into neighboring pixel, the spatial distributionof the scattered photons provides information for the reassignment ofthese scattered photons back to the correct pixel. Note that thistemporally encoded information is not available in a CCD-based MMMsystem where the temporal information is lost during the integrationprocess of the CCD. The effect of emission photon scattering on imagingcan be described as follows. Generally, an image is formed as aconvolution of source pixels and an emission point spread function(PSF_(em)). In the set up of MAPMT-based MMM, the array of 8×8 pixels iscollected together each time so that the vector of pixels are acquiredtogether, {S_(acq)} (64×1) which is the product of convolution matrix,[C] (64×64) and the source pixels, {S_(s)} (64×1),{S_(acq)}=[C]×{S_(s)}. The convolution matrix, [C] is constructed basedon the simplified PSF_(em), EPSF_(em) in which PSF_(em) is spatiallyintegrated over the effective detector area of the individual pixels ofthe MAPMT. Since EPSF_(em) has a very coarse spatial resolution of 45 μmwith the spatial integration, deconvolution with EPSF_(em) becomessimple and is less sensitive to noise. The study of emission photonscattering on PSF_(em) shows that the scattered emission photons formadditional intensity distribution around the PSF_(em), which is formedwith ballistic unscattered emission photons. Its distribution is broadwith its FWHM of 40 μm range in the imaging depth of 2×17. The change ofPSF_(em) due to scattering (FIG. 4c ) affects EPSF_(em) by increasingintensity in the neighboring pixel areas (FIG. 4d ). In the process ofdeconvolution, EPSF_(em) is roughly estimated by measuring the intensityratio of the real image to the ghost images as a function of imagingdepth. The convolution matrix, [C]_(esl) is constructed based on theestimated EPSF_(em). The source pixel vector, {S_(s)}_(esl) is acquiredby the product of the inverse transform of [C]_(esl) and the acquiredpixel vector, {S_(acq)}.{Shd s}_(esl) =[C] _(esl) ×{S _(acq)}  (2)

The restored image is presented in FIG. 4(b). The signal decay of adepth sequence of restored images is measured and the decay coefficient,c is significantly reduced to 1.58 after the deconvolution algorithmbecause the scattered emission photon can now be corrected andreassigned. The ghost images are almost completely eliminated as aresult. Restoration algorithms can be further refined such as by addingmaximum likelihood estimation to minimize image structural overlapbetween neighboring pixels. This simple deconvolution approach improvesvery effectively the performance of MAPMT-based MMM and allows thissystem to perform within a factor of two compared with conventionalmultiphoton microscope.

The performance comparison of the two MMM systems can also be evaluatedfor the imaging of biological tissues. The specimen is an ex-vivo braintissue section with neurons expressing green fluorescent protein (GFP).Thy1-GFP transgenic mice are deeply anesthetized with 2.5% Avertin(0.025 ml/g i.p.) and transcardially perfused with PBS, followed by 4%paraformaldehyde. Brains are dissected and placed overnight in cold 4%paraformaldehyde. 1-mm thick coronal sections are taken by vibrotome,mounted and coverslipped on microscope slides using adhesive siliconeisolators (JTR20-A2-1.0, Grace Bio-Labs, Bend, Oreg.). The specimen isimaged in 3D with both CCD-based MMM and a MAPMT-based MMM. Theobjective used is 20×water immersion with NA 0.95 (XLUMPLFL20XW,Olympus, Melville, N.Y.). The input laser power is 300 mW at 890 nmwavelength. The frame rate is 0.3 frames per second with 320×320 pixels.The slow frame rate is set in order to collect enough emission photonsup to 120 μm deep. The total imaging depth is 120 μm with 1.5 μm depthincrement. Representative images are shown in FIG. 5. The first columnof images are from CCD-based MMM at surface, 30 μm, and 75 μm deep. Thesecond column of images are the ones from MAPMT-based MMM, raw imagesand the third column are after deconvolution processing. On the surface,the dendritic structures of neurons are visible in all images. However,the image from CCD-based MMM does not provide as good contrast ofneurons as MAPMT-based MMM. This is because some of the emission photonsthat are initially forward propagating into the tissue are eventuallybackscattered. These backscattered photons are acquired in the incorrectpixels of the CCD and degrades the image SNR. Starting at about 30 μm,background noise increases and thin dendrite structure becomes invisiblein CCD-based MMM images. On the other hand, in the images fromMAPMT-based MMM, dendrites are still visible due to lower backgroundnoise and higher SNR. In the image of 75 μm deep from MAPMT-based MMM,ghost images of a bright cell body appear in the neighboring pixels. Theghost images are restored to the original image after the deconvolutionprocess is applied. And also it is noted that the intensity of theoriginal image is increased.

However, additional improvements of this system can be made. First,since the MAPMT is positioned in the image plane, the location of eachexcitation focus corresponds to the center position of the matchingpixel of the MAPMT. The effective detector area scales quadraticallywith the separation of the foci. Therefore, with wider foci separation,the MAPMT has higher collection efficiency for scattered emissionphotons. In the current configuration, the excitation foci are separatedfrom each other by 45 μm so that the effective detector area for eachchannel of the MAPMT is 45 μm×45 μm. The size of imaging field with 8×8foci becomes 360 μm×360 μm. As the excitation foci are separated more,the system becomes less sensitive to the scattering of emission photons.The maximum separation of excitation foci is limited by either the fieldof view of the objective or apertures of other collection optics. The20×water immersion objective used has the field of view of 1000 μm indiameter. This allows positioning the foci as far apart as almost 100microns in this example.

A limitation of the MAPMT-based MMM system compared with a CCD-based MMMdesign is that the signals are de-scanned. In the de-scannedconfiguration, emission photons are processed by more optical elementsincluding the scanner mirror before they are collected at the MAPMTsuffering more optical loss at each reflection. Further, the de-scannedgeometry also has a longer optical path that contributed to the loss ofsome scattered photons due to the finite aperture of the optics. Thesignal collection efficiency is approximately 70% in this example due toadditional optical elements. An MAPMT-based MMM system in anon-de-scanned geometry for example can recover this loss.

The MAPMT is manufactured with a current quantum efficiency of about 20%compared to 80% quantum efficiency of the CCD camera. However, MAPMT hasvery low noise. It has 20 dark counts per second without cooling and canbe several orders of magnitude lower with cooling. Since the MAPMT has areadout rate of approximately 20 KHz, the typical dark count per pixel.is less than 1×10⁻³. In comparison, the CCD noise is dominated by bothread noise and dark noise which are a few counts per pixel. Therefore,for very low photon count situation, i.e. dim sample or high frame rate,the MAPMT system can have superior performance. MAPMTs with highersensitivity cathode materials such as GaAsP can provide a system with aquantum efficiency up to about 40-50%.

The photon sensitivity of each channel is not equal and can vary up to50%. This effect is further compounded by the Gaussian geometry of theexcitation beam which results in higher excitation efficiency at thecenter pixels verses the edge region. This problem has been solvedpreviously using multiple reflecting beam splitter to generate equalintensity beam-lets. The MAPMT-based MMM system can be further improvedby utilizing this type of beam splitter with an additional flat fieldcorrection algorithm to remove inherent sensitivity non-uniformity ofthe MAPMT.

There is also cross talk between neighboring pixels of MAPMT. Thetypical crosstalk is minimal at about 2% when the photons are collectedat the center of each pixel. However, this cross talk can be removed bypost-processing of the image similar to ghost image removal discussedpreviously.

In MMM imaging, more power of excitation light is required. Assumingthat input power of 10 mW is needed for each excitation focus,generation of 64 excitation foci requires 640 mW input power. In theimaging of turbid tissue specimens, more input power is required tocompensate the signal loss due to excitation photon scattering. In caseof a tissue specimen whose mean free path length is 160 μm at excitationwavelength, the input power of 2200 mW is required to image at 100 μmdeep, assuming that signal level is decreased only due to excitationphoton scattering and there is no change in collection efficiency ofemission photons. Therefore, the current power of Ti-Sapphire laser islimited for MMM imaging and can further increase in imaging speed by theuse of even more foci.

Referring to FIG. 6, a preferred embodiment provides for parallelizedillumination and detection device which uses a common focusing device,such as an objective lens. The device provides simultaneous measurementof intensity, lifetime, spectroscopic or other information from thefocal spots (foci 151, 152, 153). Light from a first illumination lightpath 141, a second illumination light path 142 and a third illuminationlight path 143, which paths present to each other at relative angles,enter a common focusing device 110 (such as an objective lens). Thefocusing device 110 generates from each illumination light path 141,142, 143 a separate intensity cone. A first detector 121, a seconddetector 122 and a third detector 123 detect light generated by theintensity cones associated with each first, second and thirdillumination path, respectively. Light from a first illumination path141 illuminates the focus spot 151 in the sample 105, with the detectedlight following a first illumination and detection light path 161 andfirst detection light path 111 to reach the first detector 121.Similarly, light from second and third illumination paths 142, 143illuminate the focal locations 152 and 153, respectively, in the sample120, with the detected light following second and third illumination anddetection light paths 162 and 163 and second and third detection lightpath 112 and 113 to reach the second and third detectors 122 and 123,respectively.

In the multi-photon case, light from each path generates a 3D intensitydistribution in its associated focus, according to the multi photonexcitation process. The detectors 111,112,113 detect all the light inthe ‘detection cone’ associated with their active area. This lightincludes light generated by the light path associated with each detector(for example, light from the first focus 151 is detected by firstdetector 121), as well as light that is generated in the first focus 151but scatters around the first focus 151 on its way to the first detector151, and light that is generated in the second and/or third foci 152,153 and is then scattered into the detection cone of the first detectionlight detection path 111.

In the confocal case, a confocal pinhole is placed in front of thedetectors, for instance, in FIG. 6 a confocal pinhole can be placedbetween each detector 121, 122 and/or 123 and the associated reflectorsand collimation lens 126, 127, 128, 131,132 and/or 133, respectively. Asa consequence of the pin-hole impeding much of the scattered light,according to the confocal principle, only light from the focal spotassociated with that detector is collected in each detector. Forexample, following the first light path, only light from the 3D lightdistribution in the first focus 151 is detected by detector 121. A setupcould as well consists of a mixture of detectors with and without aconfocal pinhole.

In order to reduce cross talk between the light beams due to scattering,the illumination light and the associated detection can be timemultiplexed.

Still referring to FIG. 6, a device according to the invention canbecome an imaging device by the illumination beams 141,142,143 beingangle-scanned with respect to the focusing device 110. The imaging ofthe x-y planes is enabled by rotating the device through twoperpendicular angles theta and phi around the x and y axes,respectively. The intensity information is recorded along with theangular position of the device and reconstructed by a image processor.Imaging of zy planes can be achieved as well by scanning the sample inrespect to the imaging device in xy. In an imaging mode, in which thebeams are scanned, the device is capable of simultaneously generating 2Dimages of sub-regions of samples. By simultaneously imaging with eachseparate illumination and detection pathway, the speed at which imagesare generated can be increased by the number of illumination paths anddetection channels.

Imaging in the z plane occurs by moving the imaging device with respectto the sample, or vice-versa. The intensity information is recordedalong with the z-position of the sample or device and reconstructed byan image processor.

Another embodiment according to the invention provides for a multifocal,multiphoton microscope based on MAPMT, as illustrated in FIG. 7, inwhich an expanded excitation beam 104A comes from bottom of FIG. 7 andilluminates a square microlens array 140A. A plurality of opticalpathways is generated by the micro lens array 140A in conjunction withlens L1; for instance, in the embodiment illustrated here the microlensarray 140A splits the excitation beam 104A into 8×8 multiple beams(i.e., 64 beamlets). In FIG. 7, only two beamlets 141A, 142A areray-traced. A specimen 105A is scanned with an 8×8 array of excitationfoci 150A, which includes focus spots 151A and 152A illuminated bybeamlets 141A and 142A respectively. The sample area that eachexcitation focus covers can be relatively small the focus is in x and ydirection, the full width half maximum (FWHM) of the focus is 200-1000micro meter. In z direction the FWHM is 200-5000 micro meter. In animaging configuration each foci scans an area of the size of thedistance of the foci, meaning 10-1000 microns (the scanning isaccomplished by an optical scanner 180A such as, a galvo-scanner). Thetwo lenses L2 and L3 guide the plurality of optical pathways onto therear aperture of the focusing device 110A. The detection light paths,111A and 112A, respectively, resemble the illumination light path untilthe light paths are separated by a light reflector, which is in thiscase a dichroic mirror 130A. The light is then focused by a common lensL4 onto the multi anode PMT detectors 120A. The emission beam-lets arecollected at pixels 121A and 122A, respectively, of a multi anode PMT(MAPMT) 120A. The MAPMT 120A, which has the same number of pixels asexcitation beamlets, detects the signal of 8×8 pixels synchronized withthe scanning. The intensity information is recorded along with theangular position of the scanner and reconstructed by an image processor.

As shown in FIG. 8, a further embodiment according to the inventionprovides for a multifocal, multiphoton microscope based on MAPMT, inwhich an expanded excitation beam 104B comes from laser 101B andilluminates a micro lens array 140B. A plurality of optical pathways isgenerated by the micro lens array 140B in conjunction with lens L1; forinstance, in the embodiment illustrated here two beamlets 141B, 142B areray-traced. A specimen 105B is scanned with an array of at least oneexcitation foci 151B and/or 152B which are illuminated by beamlets 141Band 142B respectively. The scanning is accomplished by a scanner 180B.The two lenses L2 and L3 guide the plurality of optical pathways ontothe rear aperture of the focusing device 110B. The detection paths, 111Band 112B, respectively, depart from illumination light path whenseparated by dichroic mirror 130B. The light is then focused by a commonlens L4 and reflector 134B onto two multi anode PMT detectors 120B,124B. The MAPMT detectors 120B, 124B each detect the same number ofpixels as are emitted excitation beamlets, integrating the signal of theat least one pixel synchronized with the scanning.

A z-piezo actuator 109B (such as MIPOS 250 SG, micro-objectivepositioning system, integrated strain gauge motion: 200 μm (closedloop), Piezo System Jena controllable by controller 170B is attached tothe objective lens 110B in order to move it in the z direction for 3Dimage generation. The sample 105B is attached to a sample stage 115B,which can be moved in x, y and z directions, also controllable bycontroller 170B and/or computer 176B. Light reflector 134B (such as, forexample, a dichroic mirror) is positioned in the detection pathways toenable multi channel imaging by a first MAPMT detector 120B and a secondMAPMT detector 124B, for multi channel imaging.

An IR block filter 116B (such as e700sp Special, Multi-Photon Blocking,Block 750-1000 nm>OD 6, Chroma Technology Corp is positioned in thedetection pathway to separate the long wavelength excitation light fromthe short wavelength detection light. The filter 116B is exchangeablewith a variety of filters or can be removed completely for reflectedlight confocal imaging. The filter 116B can be mounted on a motorizedmount, which allows it to exchanged via a controller 170B and/orcomputer interface 176B. A band-pass filter 117B (such as 560DCXR fordetecting DAPI and FITC, Chroma Technology Corp) (560DCXR for thetransmission of light generated by the excitation of GFP and Rhodamin,HQ460/40 for the transmission of light generated by the excitation ofDAPI, HQ630/60 for the transmission of light generated by the excitationof Alexa 594 bandpass; Chroma Technology Corp) is positioned in front ofeach of the multi anode PMTs 120B, 124B, in order to detect certainspectra. The band-pass filters 117B, 117B are exchangeable with otherdifferent filters and can be mounted on a motorized mount enablingchanging of filters via a controller 170B and/or computer interface176B. The same sample region can then be imaged with a different set ofband-pass filters for more than two-color imaging.

A detection-part light-shield enclosure 118B is used to shield thedetection part of the apparatus from ambient light. A variable iris 119B(such as for the case of a manual version D20S—Standard Iris, 20.0 mmmax. Aperature; Thorlabs. Motorized versions of equivalent devices areavailable as well) is positioned in the focal plane of the micro lensarray 140B in order to enable single spot illumination. For 8×8 fociimaging, the iris 119B is relatively open and for fewer or single spotimaging the iris is relatively closed, so that only a few or one microlenses illuminate the sample. The variable iris 119B does not have to bein a round shape; and it will be square in shape when only a certainarray of micro lenses should be blocked to enable illumination with aview of selected foci only. The variable iris 119B can be motorized andcontrolled via controller 170B (such as, for example by connection191B), and/or via the computer 176B.

A micro lens foci mask 125B (such as a thin (for example 0.3 mm)aluminum sheet in which small wholes (for example 0.5 mm wholes) aredrilled, at the points of where the micro lens focuses) positionedproximate the micro lens array 140B is a pinhole mask with a largepinhole size that enables the transmission of most of the light focusedby the micro lenses, but otherwise it blocks ambient and stray lightfrom the laser.

A first reflector 131B generates a first laser reference beam 165B fromthe incident laser beam (for monitoring the laser illumination power,wavelength and angular pointing stability). The reference beam 165Bprojects upon the diode or detector 160B which generates a signal thatmeasures the laser illumination power, wavelength and angular pointingstability.

A further embodiment of the invention provides for a scan reference beam166B from a scan reference beam illumination source 168B to be projectedvia reflector 172B and reflector 132B onto the scan region, whereuponthe returning scan reference beam returns via reflector 132B to passthrough dichroic 172B and lens 174B to be received by detector 164B. Thescan beam is provided for monitoring the scanning accuracy. Detector164B can be a diode or CCD detector or another type detection device. Asshown in FIG. 8, an embodiment of the invention can provide for thedetector 164B to be a CCD camera, which can be used to compare imagesgenerated by CCD camera detection methods and other detection methodsaccording to the invention that employ one or more multi anode PMTs asdescribed above.

A high voltage power supply 188B supplies power to the multi anode PMTs.Multi channel photon counting cards 184B, 186B are connected to eachelement of the MAPMTs, with one photon counting device for every multianode PMT element, such as, for example, MAPMT elements 120B and 124B.

A computer 176B (including input devices, such as, for example, akeyboard and mouse) can be provided in one embodiment, connected tocomputer display 178B. The computer 176B can be connected to controller170B.

The computer 176B controls numerous elements of the invention eitherdirectly and/or indirectly through controller 170B, and one skilled inthe art will appreciate that numerous alternative configurations can beimplemented within the scope of the invention.

One embodiment provides for the computer 176B to be programmed with aprocessing software and for the computer 176B to control a number ofoptical elements through a variety of electronic interfaces. Forexample, without limitation, the computer 176B and/or the controller170B can be electronically interfaced with the scanner 180B and themulti channel photon counting cards 184B, 186B to perform the steps ofscanning and data acquisition. Further the computer 176B can performimaging post-processing steps. The display 178B can be used to displaythe acquired images in real-time after further processing.

A laser power attenuator 163B can be provided to control the laserincident power. The attenuator 163B can be controlled by the controller170B and/or by the computer 176B in order to enable power adjustmentsfor different samples and different locations in samples. During imagingat different depths in the sample, for example, the laser power can beautomatically adjusted, so that the laser power can be increased athigher penetration depth. The attenuator 163B is integrated in order tomake laser power adjustments, such as, for example, low power at thesample surface and increased power at increased penetration depth.

A third reflector 133B generates a second laser reference beam 167B fromthe incident laser beam (also for monitoring the laser illuminationpower, wavelength and angular pointing stability). This second laserreference beam 167B projects upon a second diode or detector 161B togenerate a signal that measures the laser illumination power, wavelengthand angular pointing stability. A laser power attenuator 163B controlsthe laser incident power and is integrated in order to make laser poweradjustments, such as, for example, low power at the sample surface andincreased power at increased penetration depth. Laser 101B is anillumination light source, such as a titanium sapphire laser (Mai TaiHP, Spectra Physics).

Multi-photon microscopy works most efficiently with short laser pulses.Owing to dispersion, the optical elements in the illumination pathwaybroaden the initially short laser pulse. The pulse compressor 102B isbuilt from a pair of standard high reflectance mirrors and a pair ofprisms (IB-21.7-59.2-LAFN28, Material: LaFN28; CVI Laser Corp.,Albuquerque, N.M. 87123) mounted on translational and rotational stagespre-chirps the laser pulse in order to attain a short laser pulse in thefocus of the objective lens.

A confocal pinhole array optionally can be placed between either of themulti anode PMT arrays 120B, 124B and the band-pass filters 117B, 117B,respectively. This option enables the system to be used for confocalmicroscopy or for multi-photon microscopy with confocal detection.

A telescope 103B expands the laser beam. With different expansionratios, a different number of micro lenses can be illuminated. With asmall beam expansion for example, a relatively smaller array of 2×2micro lenses can illuminated and, thus, an array of only 2×2 foci isgenerated. As the beam expansion is made larger, an array of 8×8 or moremicro lenses can be illuminated and, thus, an array of 8×8 or more fociis created. A further preferred embodiment employs a set of at least twomirrors 135B, 136B after the telescope 3B for precise beam alignment.

A mechanical micro lens holder 145B enables the precise positioning ofthe micro lens array 140B with respect to the multi anode PMTs 120B,124B in the x, y and z directions. The holder 145B can be a motorizedholder and can be controlled through a computer interface 176B, or,alternatively, can be controlled via a controller 170B, which controllerin turn can be directed by computer 176B.

Mechanical multianode PMT holder 125B, 126B enables the precisepositioning of the multi anode PMTs 124B, 120B, respectively, withrespect to the micro lens array 140B in the x, y and z directions. Theholders 125B, 126B can be motorized holders and can be controlledthrough a computer interface 176B, or, alternatively, can be controlledvia a controller 170B, which controller in turn can be directed bycomputer 176B.

The computer 176B, or the controller 170B, or the computer andcontroller together can be configured to control automatically or tocontrol in supervised fashion, one or more of the following elements,without limitation: the scan reference beam illumination source 168B,the sample piezo stage 115B, the objective z-piezo stage 109B, the scanreference beam detector 164B, the scanner 180B (by connection 193B), theIR block filter 116B (by connection 194B), the band-pass filters 117B(for example, by connection 195B), the laser source 101B, the laserattenuator 163B, the first laser reference beam detector 161B, thesecond laser reference beam detector 160B, the pulse compressor 102B,the multi-photon channel counting cards 184B, 186B, the mechanicalmulti-anode PMT holders 125B, 126B (for example, by connection 190B),the variable iris 119B (such as, for example, by connection 191B), andthe mechanical micro lens array holder 145B (for example, by connection192B).

The focal region has a focal pattern variation in xy plane. The foci canbe distributed unevenly, e.g., the rows and columns do not have to bespaced uniformly.

Also, a system can be built, in which there are additional rows and/orcolumns of PMT's at the outer region of the array of detection tubes.For example, there can be more than 8×8 rows and columns in both themicro lens array and the detector. This is particularly important fordetecting scattered photons of the outer foci and for using theinformation of the scattered photons from the outer foci fordeconvolution purposes.

Further, an embodiment of the invention provides for a system in whichthere are more detector elements than there are foci, so that aplurality of detector elements (or detection pixels) collect the photonsof one, optically conjugated foci. For example, a 16×16 detector arraycan used as a detector device, while an array of 8×8 foci can beilluminated by an 8×8 multi lens array. Smaller and larger PMT-to-fociratios can be utilized. In particular, a detector array in which onefoci is optically conjugated to an uneven number of detector elementscan be employed. This is important for detecting scattered photons inthe neighboring channels of the to the focus optically conjugateddetector and for using the information of the scattered photons fordeconvolution purposes.

The image of the sample is formed by scanning in the optical plane (xy)when the intensity signal from the detectors is correlated with the focipositions. The foci scan the specimen in the x direction, then move anincrement in the y direction, and then raster in the x direction againuntil the sample is fully covered at some desired resolution. During therastering, intensity light signals are recorded by the multi anode PMT.These signals are then saved along with the foci positions in thecomputer and can be concurrently or afterwards displayed by the computerdisplay or other graphics outputs. The foci positions are known by thescanner position (beam scan) or the sample position (stage scan). Thesmaller the step increments, the higher the resolution the final imagewill be. The scanning can be performed in a raster fashion, or in manyother ways, such as with time multiplexed methods, or scanningsimultaneously at different depths.

Referring to FIG. 9, an embodiment of the invention provides forgenerating and detecting a 3D foci pattern in focal region 154C. Asource of light is directed onto a micro lens array 140C and a pluralityof optical pathways is generated by the micro lens array 140C inconjunction with lens L1; for instance, in the embodiment illustratedhere, the microlens array 140C splits the excitation beam 104C into 8×8multiple beams (i.e., 64 beamlets). In FIG. 9, only one beamlet 141C isray-traced. A specimen 105C is scanned with an 8×8 array of excitationfoci, which includes focus spot 151C illuminated by beamlet 141C. Thescanning is accomplished by an optical scanner 180C. The two lenses L2and L3 guide the plurality of optical pathways onto the rear aperture ofthe focusing device 110C. The detection light path, 111C, resembles theillumination light path until the light paths are separated by a lightreflector, which in this case is a dichroic mirror 130C. The light isthen focused by a common lens L4 onto the multi anode PMT detector pixelelement 121C. Changing the focal length or the positions of the microlenses of the microlens array 140C with respect to each other generatescollimated and non-collimated beams at the back aperture of theobjective lens 110C. These beams generate a 3D pattern of foci. The 3Dpattern of foci generates light which is collected by the detectorarray. According to the positions along the optical axis of the microlens array, the positions of the PMT's are changed. In case ofone-photon illumination, such as is illustrated in the “Option I”detection region 124C, no confocal pinholes are placed in front of thedetectors. In an alternative embodiment, such as is illustrated in the“Option II” detection region 126C, a plurality of confocal pinholes 128Care placed in front of the plurality of detectors cells. Each of thedetection options 124C, 126C can be used for single photon and/or formulti photon imaging. The MAPMT, which has the same number of pixels asexcitation beamlets, integrates the signal of 8×8 pixels synchronizedwith the scanning, although other array dimensions can also be used.

FIGS. 10(a)-(i) illustrate in greater detail the arrangement andprogression of foci corresponding to the relative shifting in positionof micro array lenses and MAPMT pixels shown in FIG. 9. FIG. 10(a) showsan expanded detail of the focal region with 3D foci pattern. FIG. 10(b)illustrates an array of excitation light beams (in this case, an arrayof 2×8 beams) illuminating a focusing device 110D, such as, for example,an objective lens, as viewed here in the x-z plane. In accordance withdiffering degrees of collimation of at least two of the light beams,focal points 151C and 152C are created for the two beams at certaindistances, d₁ and d₂, respectively, along the optical axis (z-axis).According to the relative angle of the illumination light beams withrespect to the optical axis, the array of foci are separated in theoptical plane. Controlling parameters of the beams provider selection ofa variety of 3D foci distribution(s). For collimated light, theexcitation foci 151C is at a distance from the focal objective, f_(obj),designated here as distance d₁. For a second beam that is not collimatedperfectly, an excitation foci 152C is at a distance d₂ that is not equalto the focal objective, as depicted in FIG. 10(b). FIG. 10(c), depictinga “static” view of an x-y plane “slice” at focal depth d₁, illustrates afirst row of 8 foci (of the 2×8 array in this example) all lying at thesame focal depth d₁, understanding that any one of these foci maycorrespond with the 151D focus point in the x-z plane view of FIG.10(b). Similarly, FIG. 10(d), depicting a “static” view of a second x-yplane at focal depth d₂, illustrates a second row of 8 foci (of the 2×8array) all lying at the same focal depth d₂, any one of which foci mightcorrespond with the 152C focus point in the x-z plane view of FIG.10(b). FIG. 10(e) illustrates the separation of the two x-y planes asviewed in the y-z plane, where it can be seen that the two rows of fociare separated by a difference in focal depth D=d₁−d₂. FIG. 10(f) shows a3D, 8×8 beam matrix (64 beams) of excitation foci that have beengenerated. Each row of 8 foci lies in a different z plane, as depictedin a view of the same set of 64 foci as seen in the y-z plane (FIG.10(g)); this is a graphic depiction of a “still” configuration, i.e.,without the array being moved in a scanning mode. FIG. 10(h) illustratesan x/y view in a scanning configuration, where each line of foci isscanned in the xy plane to cover the whole yx image in its particularz-plane. A number of xy planes are shown simultaneously, but actuallyeach plane lies at a different focal depth on the z-axis. FIG. 10(i)provides a view of a section in the yz plane, illustrating the scanningconfiguration while the z/y scan is performed, i.e., scanning along they axis and through multiple depth layers in z. As a result, a 3D volumecan be imaged by only scanning the foci array in xy. Note that the x/y,x/z and y/z coordinates illustrate the associated planes; they can bedisplayed with an arrow in their positive or negative direction.

3D AM-PMT MMM can be used in multi photon endoscope device in accordancewith another preferred embodiment of the invention (to be added).

Referring to FIG. 11(c), a further embodiment provides for a 3Dcytometer, based on multi-focal, multi-photon microscope with a multianode PMT detector. A 10 W solid state pump laser 100D pumps a titaniumsapphire laser 101D (Millennia X & Tsunami, Spectra Physics, MountainView, Calif.), which generates maximum output power of 2.5 W at 800 nm,and 120 fs pulses at a repetition rate of 76 MHz. The light is conductedthrough two reflectors 137D, 138D and then passes through a firsttelescope 103D, two additional reflectors 139D, 133D, an attenuator163D, and a second telescope 203D. After passing through anotherreflector 136D, the light is subsequently split into an array of beamsby the micro lens array 140D and is transmitted by lenses L1, L2 and L3onto the back aperture of the objective lens 110D, thus creatingmultiple foci in the focal plane. The micro beams are scanned by axy-scanner 180D (Cambridge Technologies, Cambridge, Mass.). Thefluorescence is collected by the same lenses and separated from theillumination light by a dichroic filter 130D and a two-photon blockfilter 116D. The fluorescence passes through lens L4 and is thenseparated into two spectral channels by the dichroic filter 134D anddirected onto the multi-anode PMTs 124D, 120D. The degree of spectralseparation can be chosen depending upon the application. The embodimentdisclosed here uses a red/green and a green/blue filter to accomplishthe spectral separation. The variation of the magnification of thetelescope 203D enables the utilization of, for example, 4×4, 6×6 or 8×8arrays of micro lenses, among other size arrays.

FIG. 11(b) illustrates an image of the array of foci in the focus of theobjective lens, such image as can be taken by a CCD camera, where herethe foci are not scanning. The foci are 45 μm apart in resulting in apotential scanning field of 240 μm when 6×6 foci are utilized.

FIG. 11(c) shows a z-profile and a corresponding fit function of a 200nm bead. The system shows a resolution of 2.4 μm, which is close to atheoretical value of 2.2μm, considering the under-fulfillment of theback aperture of the objective lens. Acquisition speed for this scanningprofile was 10 frames per second. The profile is averaged over 5consecutive pixels, reducing the sampling from 30nm per pixel to 150 nmper pixel.

Referring to FIG. 12(a) a further embodiment of the invention providesfor multi color detection MMM in the xz-plane. An array of 2×8 beams isgenerated by the micro lens array 140E. The setup here is illustratedwith two 1×8 beam lines. The distance between the foci in each line isdetermined by the combination of the source beam configuration and themicro lens array 140E. Two light beams are conducted through the microlens array 140E and intermediate optics onto the focal plane of themicroscope in which they create two lines of 1×8 foci. For simplifiedvisualization, in FIG. 12(a) only 3 of the 16 beam traces in a 2×8 setupare illustrated. The full field is then scanned by the mirroroscillation of the scanning mirror 180E, in which the scanningamplitudes need to be adapted to the distances of the foci. On thedetection side a holographic diffraction grating 192E is incorporatedthat diffracts the multiple wavelengths emitted from the sample onto thephoto-multiplier arrays of two stacked multi-anode PMTs 120E, 124E. Inthe setup the two multi-anode PMTs will be stacked on top of each other,each serving as a spectral detection device for one line of 1×8 foci.The grating 192E properties (pitch/inch) and the focal length of thefocusing lens (L4), which determines the distance between the gratingand the multi-anode PMT, have to be chosen in accordance to theanticipated fluorescent probes used for staining the tissue sample. Forthis embodiment, a transmission grating 192E is used. Nevertheless,comparable and/or better efficiency can be achieved in embodiments thatuse a reflection grating or a prism. FIG. 12(b) illustrates theillumination foci and their scanning in the focal xy-plane. Scanning isindicated for two arrays of 8 foci each. FIG. 12(c) shows the detectionpath of two beams projected in the yz-plane through grating 192E andlens L4 onto the stack of two AM-PMTs 120E, 124E. FIG. 12(d) shows thedetection path projected in the xz-plane, where the beams are depictedpassing through grating 192E and lens L4 with each of eight color bandsbeing collected by the two AM-PMTs 120E and 124E. FIG. 12(e) illustratesthe anodes of the multi-anode PMTs 120E, 124E in the x/y plane, showingthat the 8×8 anode arrays of the detectors each detects one of the two1×8 beam lines, where each 1×8 beam line has been diffracted by thegrating 192E into eight color bands.

Referring to FIG. 13, a beam splitter device 400 can be used to create ahomogenous intensity profile over a plurality of beamlets. Dependingupon its design, the beam splitter splits one beam into 256, 128, 64, 36or 16 approximately equally powered beams by one or more fullyreflective or semitransparent mirrors. In FIGS. 13 50% and 100% indicatethe percent reflectance of the mirrors used, where a series of fullyreflective mirrors 420 with one longer semi-transparent 410 mirrorsplits the beam in the x-plane (BS-X). By combining two such cubes inseries, it is possible to generate a 2D array of beamlets. The internaloptics of a second beam-splitting cube for the y-plane (BS-Y), are thesame as for x-plane beam splitting cube. The beams are then focused bymicro lens 430 (or via other multifocal optics) through lens 432 andobjective lens 434 onto the focal plane.

FIGS. 14(a)-14(d) illustrate additional preferred embodiments forproviding multifocal illumination including a micro-lens array 140N fromexpanded beam 201N in FIG. 14a and FIG. 14b a diffractive opticalelement 205N separates beam 201N into a separated plurality of beamswhich are coupled to focal locations as previously described herein. InFIG. 14c a plurality of optical fibers 220N can be used to provide aplurality of beams with lens L1 for delivery to the focal locations orspots. As seen in FIG. 14d the fibers 220N can position beams indifferent directions for smaller or greater focal separation.

In addition to the primary use of this instrument for two photonmicroscopy, other multi-photon sensing and imaging methods can also beused with the system described herein including:

-   -   2, 3, or more photon excitation microscopy,    -   second, third ore more Harmonic Generation microscopy,    -   coherent anti Stokes Raman scattering (CARS) microscopy,    -   multi photon quantum dot imaging,    -   surface plasmon imaging, and    -   Stimulated Emission Depletion (STED) microscopy        With the implementation of a confocal pinhole array, shown in        FIG. 8, confocal microscopy can be preformed with the same        instrument.

FIGS. 15(a)-15(d) illustrate further preferred embodiments for use withdetectors which can be a multi anode PMT or an array of singledetectors, connected via optical fiber. The detectors can be PMT's oravalanche photo diodes, or the detector array can be a combined device(like a multi anode PMT), connected via optical fiber, an avalanchephoton diode array, a CMOS imaging detector, or a CCD camera in whicheach pixel or each area of binned pixels is correlated to one focus, ora CCD camera in which more than one pixel or more than one binned pixelarea is correlated to one focus. As seen in FIG. 15a , the detector 210Pcan be coupled directly to optical fibers 220P which receive light fromlens L1. As shown in FIG. 15b , individual detectors 210P can collect atdifferent angles, or as seen in FIGS. 15c and 15d , a detector array212P can detect at the same or different angles respectively.

Referring to FIG. 16, the optimal number of foci for a two photonexcitation process at a certain laser power for samples with differentdamage thresholds can be determined. The optimal number of foci willdepend on (i) the damage threshold, (ii) the quadratic dependency of thetwo-photon signal to the laser power, and (iii) the limited amount oflaser power. In the graph shown in FIG. 16, the laser power is limitedto 1.2 W at the sample, while the damage threshold of the sample rangescan be 10 mW, 20 mW and 50 mW. As a result, the optimal number of fociis 120, 60 and 24 respectively. In general, the appropriate power levelfor two-photon imaging is constrained by two basic boundary conditions:(a) the minimum accepted signal-to-noise ratio determines the minimumpower that can be used, whereas (b) the damage threshold of the sampledetermines the maximum power. In an MMM system, the limited laser poweris distributed over a large number of foci. The best signal is obtainedwhen the number of foci is chosen in a manner such that each of the focidelivers a power level just below the damage threshold for the sample.The relationship is illustrated in FIG. 16. It is possible to obtainless signal from the sample as more foci are used, owing to the squareddependence of signal on laser power. A judicious choice of power levelsand of number of foci must be made in order to obtain optimal results.Therefore, a preferred method and system provides for a versatile systemin which the number of foci can be varied with respect to the samplethreshold. The threshold can be different for different penetrationdepths into the sample and can therefore be adjusted by the attenuator163B.

Time multiplexed illumination and detection enable MMM microscopy withone detector only, which is gated to the excitation light pulse. In onevariation of a multi-photon MMM, the illumination light source is apulsed laser. In this case, a Ti:Sa Laser with a repetition rate ofapprox. 80MHz and a pulse width of approx. 100-200 fs as an example. Inthe standard illumination version of this MMM, all beams carry the samepulse distribution along time. As a consequence, the array of excitationfoci in the focal region is formed simultaneously. For image formation,during or after at least one pulse has illuminated the sample, the beamor sample is scanned on both axis perpendicular to the optical plane;here indicated by x≥0.

In the example of multiplexed operation 500 shown in FIGS. 17a and 17b36 detection elements are collecting light from 18 simultaneouslyilluminated spots. The delay between the illumination pulses 502 isalternated between foci shown at 504. This configuration can be imaginedto be accomplished with more detection channels per simultaneouslyilluminated foci. For image formation, during or after at least onepulse has illuminated the sample, the beam or sample is scanned on bothaxes perpendicular to the optical plane; here indicated at 506 by x≥0.Depending on the particular configuration, there are several differentadvantages. First, the light in non-corresponding detection channels hasan additional time delay relative to light from the foci correspondingto the detector channel 508 that is receiving light at a particulartime. In fast processes, the resulting signals may not or may minimallyoverlap and thus be registered to the proper foci directly. In slowerprocesses, where the overlap may be significant, the temporal separationwill aid numerical registration and deconvolution algorithms.Furthermore, when the number of detectors matches or exceeds the numberof illumination foci, the response in the neighboring non-correspondingdetectors can be used to generate additional information about thesample. The temporal delays introduced into the illumination foci meanthat this supersampling condition exists even when the number ofdetectors is the same as the number of foci.

In anther example, alternating excitation foci pattern can be detectedby a multichannel detector with smaller number of elements than numberof foci.

In cases in which the repetition rate of the laser is lower that in thecase of the Ti:Sa laser, a time multiplexed MMM illumination and adetection in a single channel can be used. In this particular case, therepetition rate of the laser is a hundred times lower than in previousexamples. In a time multiplexed version, each beam carries a pulse whichis temporally separated in regards to pulses of the other beams. In oneparticular case, they are separated evenly over the time period of onelaser repetition, so that at evenly distributed time points, one singlefoci is illuminated at a time. If a fast detector is correlated with thepulse distribution and capable of detecting each pulse separately duringthis short time period, an MMM with only one detection element can beused. This detection element has a corresponding detection area tocollect light which is generated by each individual foci during itsscan. For image formation, during or after at least one pulse cycle hasilluminated the sample, the beam or sample is scanned on both axisperpendicular to the optical plane. As a result, optical cross-talk iscompletely eliminated, as the light from the different foci is excitedand detected at different time points. Applications for this case areexcitation processes which appear instantly, like scattering effects(such as Second Harmonic Generation (SHG) or Coherent Ramen Anti Stokesscattering (CARS)). This configuration can be used suited for anon-de-scanning configuration. Other repetition and detection rates arepossible.

In the case of using a pixellated detector such as a CCD or a CMOSimager, an array of 3×3 beams, FIG. 18a , illuminates the focusingdevice, forming an array of 3×3 foci (FIG. 18b ). As seen in FIG. 18c ,the image of the scattering distribution of the foci is imaged by theCCD camera. The wide-field image is accumulated per scanned illuminationpoint. The scattering distribution of each foci can be recorded on manyCCD pixels. The illumination or sample can be scanned and the wide-fielddata can be further processed to form an image or statisticalrepresentation from many object points. This configuration can beemployed in a non-de-scanning configuration as well.

The above described systems and methods can be used for imaging of allsemi-transparent and highly scattering materials; 2D and 3D, and inparticular for imaging of human and animal tissue, cells in suspension,plants and other biological material.

The illumination can be achieved with visible light and alternated withthe MMM scanning measurement or out of band illumination light can beused and the camera measurement can be taken simultaneously with the MMMmeasurement. This configuration can be used for large field imaging,sample guided MMM measurements, conventional staining measurements, andonline MMM measurement process control, for example, bubble formationmonitoring, and laser spot diagnostics.

There is a large variety of fluorescent that can be used with variousembodiments of the invention dyes. In general they fall into twofamilies: Dyes that have to be applied to stain the tissue “from theoutside” and dyes, that are expressed from animals as proteins. Mostcommonly used dyes by external staining MitoTracker Red, DAPI, Hoechst33342, Cy2-IgG, Alexa Fluor 546, Rhodamine 123, Alexa Fluor 488,FITC-IgG, Acridine Orange, Cy3-IgG, Issamine Rhodamine,TexasRed-Phalloidin, TexasRed-IgG, Alexa Fluor 594, Propodium Idonide.Dyes genetically expressed by genetically modified animals: greenfluorescent protein (GFP) and other dyes in this family: Enhanced GFPEGFP, Yellow fluorescent protein (YFP), Enhanced YFP. Auto fluorescentimaging does not use a particular dye, but can be used as part of animaging technique.

Besides confocal microscopy (fluorescent, as well as reflected lightconfocal), these include all other multi-photon microscopy techniques,such as, 2, 3, or more photon excitation microscopy, Second (SHG), Third(THG) ore more Harmonic Generation microscopy, Coherent Anti StokesRaman Scattering (CARS) microscopy, multi photon quantum dot imaging,surface plasmon imaging and Stimulated Emission Depletion (STED)Microscopy. These techniques can be used with or without stainingmethods. The scattering techniques, such as SHG, THG, CARS are developedto be able to image without any staining involved.

FIG. 19 illustrates a probe or endoscope apparatus according to anembodiment of the invention, having a handle portion 272F and aninsertable probe portion 270F, wherein light delivered from a lightsource 244F (which can be a laser or other light source) is deliveredthrough an optical wave guide 234F (such as, for example, optical fiber,hollow fiber optics, photonic band gap fiber, or other wave guide) to anoptical connector 224F (such as, for example, a pigtail), whereupon anexpanded beam 104F passes through a lens or optionally through lens pairtelescope 103F and then through a micro lens array, or other opticaldevice that creates a plurality of optical pathways, 140F. Theillumination path can then pass through lens L1, dichroic 130F andlenses L2 and L3 onto the rear aperture of the objective 110F The beamis made to scan by scanner 180F which can tilt in the x and/or ydirections, and the return fluorescent signal is directed by dichroic130F and reflector 136F, optionally an IR block filter 116F through lensL4 and optionally a band pass filter 117F onto a multi-anode PMTdetector 120F. In an alternative embodiment, a plurality of confocalpinholes 119F are placed in front of the plurality of detectors cells.The detector 120F can be connected to a controller 170F and to an imageprocessing computer 176F. The scanner 180F can also be controllablyconnected by electrical connector 193F to a controller 170F and/orcomputer 176F. In FIG. 19, the proportions of the endoscope haverelationship to the focal distances of the micro lens array, fm, thelenses L1-L4, being f_(L1), f_(L2), f_(L3) and f_(L4), offset distancesd₁ and d₂, and the size will be related to the relative size of thevarious elements.

Referring to FIGS. 20(a)-(b) through FIGS. 24(a)-(b), in which commonelements share the same numbering between figures, the active area,relative proximate orientation of active detector elements (such as, forexample, the active area of multiple anode photomultiplier tube detectorelements), and the distance of the foci and the intermediate optics havean important relationship to the effectiveness of detecting scatteredlight from one or more light spots in a sample specimen, as explained inthe following.

FIG. 20(a) depicts one PMT and one excitation focus, and the directionof scattered light with respect to the detection light cone of theactive detection area will control whether or not the photon will bedetected. FIG. 20(a) provides an illustration of how the size of theactive detection area relates to scattered light detection from a spotcreated by multi photon excitation, as follows: An illumination lightbeam 204G coming from the left (parallel solid lines), generates a multiphoton excitation light spot 251G (so-called excitation point spreadfunction) in the sample 105G, in which the structure causes amulti-photon excitation process. Within this 3D sample region, light isgenerated according to the multi-photon excitation principle andscattered on its path (such as, for example, an auto-fluorescent tissuesample). The potential detection path is illustrated by the very thinbounding lines enclosing the stippled shaded region, which aregeometrically determined by the side boundaries of the active PMTdetection area 222G of the detector 120G. In this configuration, all thephotons that propagate within the detection cone, indicated by theshaded region, and that travel in the direction of the detector 120G,are collected in the active detection area 222G. Photons that propagatein the opposite direction, or that are scattered outside of thedetection cone defined by the optics and thus outside of the activedetection area 222G of the detector 120G, are not detected. This isdepicted in FIG. 20(b) as well, where the detection area 224G (dashedbox) in the sample focal plane 210G corresponds to the active detectionarea 222G, while a potential scattering region 254G (circle) extendsbeyond the confines of the detection area 224G. Three examples of photonpath are shown in FIG. 20(a), as follows: (1) An unscattered photon261G, traveling in the opposite direction to the incident light beamfollows a path 262G (solid line), which path 262G lies within thedetection light cone and travels towards the active detector area 222Gand is thus collected; (2) a first scattered photon 271G that isscattered within the detection cone follows a path 272G (short-dashline) and travels in the direction of the active detector area 222G andis thus detected; and (3) a second scattered photon 281G that follows apath 282G (long-dash line) which travels generally in the direction ofthe detector but does not fall into the detection light cone of theactive PMT detection area 222G, and thus it is not detected. Lightgenerated in the spot 251G is detected by the same objective lens 110G.It can as well be detected by an opposing objective lens and collectedby a detector associated with the detection area to the light spot atthe opposing side. Then also photons in the detection cone of thesecond, opposing lens, traveling towards the direction of the incidentlight into the opposing detector, are collected.

Referring to FIG. 21(a), when scattered light from one excitation foci251G (single excitation point spread function (PSF)) is detected in asetup with two PMT elements 120G, 124G, the gap 232G (also marked as“g”) between the active detection area 222G of detector element 120G andthe active area of the second element 124G will correspond to the gap236G in FIG. 21(b) between two detection areas 224G, 226G in the samplefocal plane. If the scattering region 254G around the excitation foci251G extends into the detection area 226G, then the photon 281G that isscattered beyond the detection cone for active detector area 222G ofdetector 120G can follow photon path 282G into the adjacent detector124G.

Scattered light detection from a spot 251G created by multi photonexcitation, detected by two large area detectors 120G, 124G, positionednext to each other, are separated by a distance: In this case, theunscattered photon 261G and the first scattered photon 271G are stillcollected by the active detection area 222G of the first detector 120G.The second scattered photon 281G is not lost, but is collected by thesecond detector 124G. This effect of light being scattered intodetectors other than the optically conjugated detectors is termed“optical cross talk”.

FIG. 22(a) illustrates scattered light detection with two PMTs 120G,124G and two excitation foci 251G, 252G, where again the issue of“optical cross talk” is relevant. Here a second illumination light path206G at an angle Ψ₁ with respect to the first illumination light path204G, creates a second focus 252G (excitation PSF) at a distance δ fromthe illumination light spot 251G. In FIG. 22(a), only one, unscatteredphoton 291G is illustrated in order to simplify the drawing. This photon291G originating from 252G follows optical path 292G into detector 124G(although the illustration includes a collimating lens between thereflector and the detector and a refraction in path 272G by said lens isdepicted, owing to constraints in the size of the drawing and toillustrate better the features emphasized here as aspects of anembodiment of the invention no refraction in the paths of 282G and 292Gis depicted in this illustration). Light originating from the secondlight spot 252G will be collected by the second detector 124G, but alsoby the first detector 120G as well because photons from the second lightspot 252G are similarly scattered as photons from the first light spot251G. FIG. 22(b) illustrates this by showing scattering area 256Goverlapping both the detection areas 226G and 224G (again the gap 236Gbetween the detection areas in the focal plane will correspond with thegap 232G between the boundaries of the active detector regions). As aresult, scattered light from light spot 252G is falling in the detectioncone of first detector 120G and will be detected by that detector 120G(i.e., optical crosstalk). With increased scattering of photons in thesample, the optical crosstalk increases. Samples with a low mean freepath length (MFP) for photons will induce photons to scatter more(scattering more times at equal traveling lengths); thus, samples withlow photon MFP will induce higher optical cross talk. In addition,increasing the imaging depth will raises the probability that a photonwill be scattered on its way through the sample to the detector, becausethe traveling length in the high scattering media is longer. Becauselight with longer wavelength is scattered significantly less,applications with relatively longer detection wavelengths have oftenbeen preferred, in order to reduce the effect of optical cross talk.However, preferred embodiments of the invention provide methods forreducing optical cross talk without having to shift to longerwavelengths.

Referring to FIG. 23(a), for instance, the invention provides forreducing optical cross talk by increasing the gap distance between theexcitation foci, this gap distance depicted as δ, and simultaneouslyincreasing the gap 232G between the active detection areas in thedetection elements. In this case, the second illumination light path206G is separated further by an angle Ψ₂>Ψ₁ from the first light path204G, generating an illumination light spot 252G in a location that islarger distance Δ>δ from the illumination light spot 251G. The detectors120G, 124G are also separated from each other by increasing gap 232Gbetween the active detector areas to a value “G”, where G>g, such thatthe second unscattered photon path 282G no longer falls into the seconddetector 124G. FIG. 23(b), illustrates this by showing no overlapbetween scattering regions 254G, 256G and the neighboring detectionareas 226G, 224G, respectively. With this configuration, optical crosstalk is reduced because fewer photons end up in the “wrong” channel;however, some scattered photons will not be detected because their pathswill pass between the active detection areas of more widely separateddetectors. Thus, detection light (signal) is lost.

Referring to FIG. 24(a), a preferred embodiment of the inventionprovides for reducing optical cross talk without inducing signal loss,by increasing the distance between the excitation foci andsimultaneously increasing the active detection area of the detectorelements. By separating the foci and the associated detectors, theoptical cross talk is reduced. By increasing the active area of thedetectors, most scattered photons are collected. In FIG. 24(a) this isdepicted by the second scattered photon path 282G being collected by itscorresponding detector 120G. FIG. 24(b) shows the expanded detectionareas 226G can encompass the scattering

In a further embodiment, changing the optical configuration of theapertures and focal length of the lenses in the optical system cancreate the same effect. Changing the aperture and the focal length ofthe micro lens array, increasing the area of the scan mirror, changingthe aperture and the focal length of the lenses L1, L2, L3 and L4 has asimilar effect of reducing cross talk without loss of signal. An examplecase is presented in tabular format in FIG. W-22(a)-(e).

The optical configurations for two alternative embodiments of theinvention, Example A and Example B, are presented in FIGS. 25(a)-(e).FIG. 25(a) lists the objective lens specifications, which are the samefor both Examples A and B (i.e., Olympus, 180 mm tube lens, XLUMPLFL 20×magnification objective; water immersion; 0.95 numerical aperture; 17.1mm back aperture; 2 mm working distance; 9 mm focal length; 22 mm fieldnumber; and 1.1 mm corrected field).

FIGS. 25(b)-(c) list the details of the illumination path. A differentmicro lens array is described for each embodiment, but in both the microlenses are square shaped.

In Example A, the side aperture pitch of each micro lens is 1.06 mm andthe diagonal is about 1.5 mm. For the entire array in Example A the 8lenses per side create a side aperture of 8.48 mm and a diagonalaperture across the array of 11.99 mm. The focal distance of each microlens is 17 mm in Example A. In the embodiment of Example B, the apertureor pitch of each micro lens in the array is 1.9 mm and its focaldistance is 25 mm.

The focal lengths of the lenses L1 and L4 are 50 mm and 103.77 mm,respectively, in Example A, while in Example B they are 40 mm and 46.32mm, respectively. When standard optical components are used, they canapproximate components with a focal distance of 100 mm and 45 mm for L4in the configurations A and B, respectively. In both the embodiments ofExample A and B, the focal lengths of lenses L3 and L4 are 30 mm and 125mm, respectively. The diameter of illumination of the back aperture ofthe objective lens for both Examples A and B remains approximatelyconstant at 13.0 mm and 12.7 mm, respectively. This results in an‘under-illumination’ of the back-aperture of the objective lens whichhas a back aperture of 17.1 mm in diameter. This is desirable, so anoptimal (maximal) employment of the illumination light power iswarranted.

As listed in FIG. 25(d), the two embodiments, Example A and B, achievedifferent distances between excitation foci in the optical plane:Example A has a foci distance of 46 microns, whereas Example B has afoci distance of 103 microns. The total optical field, listed below thefoci distance, results from the fact that in this particular case, an8×8 configuration of foci is chosen. It has a square side of 366 micronsfor configuration A and 821 microns for configuration B, when the fociare scanned. FIG. 25(e) lists the details of the detection path for eachexample. These alternative examples, A and B, can be created accordingto the layouts of either FIG. 7 or FIG. 8, according to embodiments ofthe invention. In both cases the size of the MAPMT can remain constantat about 2 mm×2 mm. Employing a MAPMT with larger detection elements canincrease the detection efficiency.

FIGS. 26(a) and (b) illustrate the foci distribution in the focal planeof the objective lens for the embodiment Examples A and B, respectively.The conjugated detection area of each channel of the multi anode PMT isby a factor of 5 larger in the Example B than in Example A.

The optimal distance between the foci is influenced by three factors:(1) the optimal number of foci that are needed to generate as much lightas possible (this number can be distinguished in accordance with thegraph in FIG. 16; (2) the corrected field of the focusing device, suchas an objective lens (the larger the corrected field, the further thefoci can be separated from each other and the more the optical crosstalk can be reduced); and (3) the numerical aperture (NA) of theobjective lens for high resolution imaging (the larger the numericalaperture of the objective lens, the more photons can be collected andthe better the images are). Nevertheless, there is a compromise betweenthe NA of the lens and its effective field of view. Therefore, theobjective lens used in a most preferred embodiment of the invention hasa large NA of around 1.0 or greater and is capable of imaging a largeeffective field of view, preferably of approximately 1-6 mm.

At a fixed number of foci, a large field objective provides an advantagefor certain embodiments of the invention, because the foci can befurther separated. An objective lens with large field of view enableslarge separation of foci and thus reduces optical crosstalk. In FIG.26(c) and (d) two objective lenses with different fields of view areshown, 600 micron objective field versus 6000 micron objective field,respectively. The conjugated detection area of each channel of the multianode PMT associated with the focal plane within this field of view is afactor of 100 larger in the objective of FIG. 26(c) versus FIG. 26(d).Commercially available objective lenses with a large numerical aperture(NA) of around and above 1 usually have a field of view for which theyare corrected between (100×objectives) around 200 mm and (20×objectives)1000 mm. With the Olympus XLUMPLFL 20X water immersion objectivementioned above and used in embodiment Examples A and B, when an arrayof 8×8 foci is employed, the optimal distance between the foci is 111.11microns and the total field imaged is approximately 1000 microns.

In the embodiments shown in FIGS. 7 and 8, the active detection area ofthe different detector channels in the MAPMT is approx. 2 mm and limitedby the commercially available MAPMT devices. If this area is increases,the optical cross talk and the collection light efficiency is increased.

One embodiment of a method for data post-processing according to theinvention is illustrated in FIG. 27(a) and provides for datapost-processing starting at a step 310H. Step 310H can includeinitiating a computer program and/or software application automaticallyas part of a data acquisition step in a computer that is connecteddirectly to the imaging apparatus and/or can include a series ofhuman-supervised data-processing steps. The data processing can beautomatically intiated by the computer and proceed entirelyautomatically according to a data-processing control softwareapplication and/or the program may proceed semi-automatically withopportunities for human supervision and intervention in one or more ofthe data processing steps. An embodiment of the data-processing methodfollows the start step 310H with a next step to load the image data312H. This can include accessing raw data and metadata from storagedevices, where metadata (data about the data) includes, inter alia andwithout limitation: foci number, pixel dimensions; pixel spacing;channels; instrument parameters (including, without limitation, optics,objective, illumination, wavelengths, beam-splitting, phasing, polarity,light pumping, pulse compression, chirping, upconversion, dispersion,diffraction, source-light properties, source light stability, sourceattenuation, reference scanning, micro lens configuration andproperties, focal lengths, filter types and positioning, detectionconfiguration, detector type, detector active area, detector sensitivityand stability, and other detector specifications and properties, interalia); sample properties and sample information, such as, for example,for biological samples (including biological and non-biologicalinformation, such as, for example, tissue type, specimen type, size,weight, source, storage, tracking, scattering properties, stain/dyetype, specimen history, and other physical properties of the specimen)or sample properties and sample information for chemical and/or physicalmaterial samples; and scanner data, including, without limitation,scanning type, scanning mode, scan method, tracking, frequency,certainty, precision, scan stability, resolution and other informationabout the scanning. Data can be stored as XML, text, binary or in anyfashion, in electronic form and/or in retrievable and scannable physicalformats. In one preferred embodiment of the data post-processing method,the next step 314H is deconvolution of the image data, whichdeconvolution is described further below. Following this, the data canbe saved in an optional step 316H, whereupon the post-processing canoptionally be stopped 318H. An embodiment also allows the processing tocontinue to a next step 320H that comprises performing an intensitynormalization on the data, which normalization steps are described inmore detail below, then optionally saving the data (step 322H), andstopping the data processing sequence (step 324H).

Referring to FIG. 27(b), another embodiment of the invention providesfor additional and/or alternative method(s) for data post-processing, Inone embodiment, the processing Alternatively, an embodiment of themethod The post-processing steps can include a number of substeps. Step332H can include accessing metadata from a storage device, includinghere by reference all the description of possible metadata describedabove for the steps illustrated in FIG. 27(a). Step 334H can includenormalizing, filtering (de-noising), and blending6 (integrating) ofmultifoci subimages, and further can include registering subimages intoa single image. Step 336H can include filtering and normalizing imagesproduced from corrected subimages. Step 338H can include registering,building mosaics, and blending sets of corrected images into a largerwhole. Also, optionally, at this step 338H, an embodiment of the methodof the invention provides for creating lower resolution images of thelarger image to facilitate access, as well as images from differentperspectives (such as, image views taken of the xy-, xz-, and/oryz-planes) and creating data-compressed versions of the data and/orresults (e.g., JPEG, wavelet compression, inter alia withoutlimitation). Step 340H can include segmenting images into objects, whichsegmentation step can either be manual, automated or a combination ofboth. Step 342H can include parameterizing the objects, samples orspecimens (such as, for example, size, shape, spectral signature). Step344H can include classifying objects into higher orderstructures/features (e.g. material stress or cracks, vasculature,nuclei, cell boundaries, extra-cellular matrix, and location, interalia, without limitation). Step 346H can include statistically analyzingparameterized objects (such as, for example, by correlation methods,principal component analysis, hierarchical clustering, SVMs, neural netclassification, and/or other methods). Step 332H can include presentingresults to one or more persons on one or more local or distant displaydevices (examples include: 3D/2D images, annotated images, histograms,cluster plots, overlay images, and color coded images, inter alia).

The post-processing steps can include a number of substeps, including,among other steps those illustrated in FIG. 27(b), without limitation:

i) after data access 332H normalizing, filtering (de-noising), blending(integrating) of multifoci subimages 334H;

ii) Registering subimages into a single image;

iii) Filtering and normalizing images produced from corrected subimages336H;

iv) Registering, building mosaics, and blending sets of corrected imagesinto a larger whole 338H;

v) Optionally, at this stage, lower resolution images can be created ofthe larger image to facilitate access, as well as images from differentperspectives (xy, xz, yz).

vi) Data-compressed versions (e.g., JPEG, wavelet compression, interalia without limitation) can be produced;

vii) Segmenting images into objects 340H This segmentation can either bemanual, automated or a combination of both.

viii) Parameterizing the objects 342H (for instance, size, shape,spectral signature).

ix) Classifying objects into higher order structures/features 344H (e.g.material stress or cracks, vasculature, nuclei, cell boundaries,extra-cellular matrix, and location, inter alia, without limitation)

x) Statistically analyzing parameterized objects 346H (e.g., bycorrelation or other methods).

xi) Presenting results to user on display device 348H (examples include:3D/2D images, annotated images, histograms, cluster plots, overlayimages, and color coded images).

FIGS. 28(a) and(b) relate to image normalization. The multi foci powermap of the MMM when a micro lens array is implemented. The numbersresemble the power in each foci in the sample. Due to the Gaussian beamprofile in an example, 51.2 mW are inhomogeneously spread over the 36foci. 24.1 mW contribute to the foci lying beyond the 6×6 foci matrixand are thus lost.

The normalized signal distribution resembles the normalized power mapsquared and shows an intensity drop of 45% toward the corner PMTs inrespect to the center PMTs. The laser power was attenuated to 75.3 mW inthe sample and can reach a maximal value of approx. 645 mW, resembling apower of approx. 15 mW for the center foci in the sample. Measuredintensity profile can be generated by imaging a homogeneouslydistributed fluorescent dye under a cover slip. The intensitymeasurement is not only mapped by the power/intensity distribution ofthe foci, but also by the sensitivity of the detector array. As a resultit resembles the “true” measured intensity distribution. The imageconsists of 192×192 pixels and was generated by an array of 6×6 fociwhich were scanned across a uniform fluorescent dye sample. 2D xy imagenormalization is carried out in different ways:

-   Case 1: The normalized inverse of this intensity image (from a    uniform fluorescent dye) is multiplied with the yx images taken of    the sample. The resulting images are then displayed and saved as a    normalized image.-   Case 2: A large number of images from a sample at various positions    (and thus with a random underlying intensity structure) is averaged.    This image is then inversed and normalized. This image is multiplied    with the original data is then displayed and saved as a normalized    image.-   Case 3: A simplified image is generated which consists of 36    sub-images (generated by the 6×6 foci). Each of the sub-images    carries the average intensity generated by the specific foci. For    example, all 32×32 pixels in the top left sub image carry the same    number; 45. The image is then inversed and normalized. This image    multiplied with the original data is then displayed and saved as a    normalized image. An image can be generated either from the    intensity image generated by the process of case 1 (fluorescent    image) or case 2 (over many images averaged). 3D xyz image    normalization is carried out in a similar fashion as in case 2 of    the xy image normalization. A z-intensity profile (an example is    FIG. 28b ) is generated by averaging the intensity signal the xy    planes form different positions in z. As the penetration depth    increases, the average intensity decreases along the z-axis. In    order to get a good average intensity for the z-intensity profile,    images from a sample at various positions (and thus with a random    underlying intensity structure) are averaged. This z-intensity    profile is then inversed and normalized. Each image plane is then    multiplied with the according normalization number generated by this    process.

A method for multifocal multiphoton imaging of a specimen in accordancewith a preferred embodiment of the present invention:

-   -   (0) Start    -   (1) Sample pre processing (optional)    -   (2) place the sample in the region of focus of the focusing        device (objective lens)    -   (3) determine imaging parameters    -   (4) set imaging parameters    -   (5) image    -   (6) Process images for feedback purposes (optional)    -   (7) display the images (optional)    -   (8) save the data    -   (9) process the data (optional)    -   (10) save the processed data (optional)    -   (11) display the processed data (optional)

Concerning the order of the steps, (1) and (2) can be switched: (4), (5)and (6) can be switched.

In more detail these are

(1) Sample pre processing

-   -   Apply tissue staining    -   Apply optical clearing agents

(2) place the sample in the region of focus of the focusing device(objective lens):

-   -   Determine desired sample region    -   Place the sample in the region of focus of the focusing device        (objective lens):        or    -   Place the region of focus of the focusing device (objective        lens) on to the sample

(3) determine imaging parameters

Determine imaging parameters by which include variable or multiplevalues for each of the parameters

-   -   measurements performed manually or in an automated fashion        outside or within the imaging procedure comprising        -   the region of interest            -   Sample shape            -   Sample margins            -   Emission intensity            -   Emission wavelength            -   Emission polarization            -   . . . *        -   sample            -   type            -   photons mean free path            -   power threshold            -   sample fixation            -   sample labeling            -   . . .*        -   illumination        -   detector        -   . . . *        -   The imaging parameters comprise            -   illumination wavelength            -   illumination power            -   illumination polarization            -   scanning speed            -   maximal penetration depth            -   sampling            -   . . . *

(4) set imaging parameters

A computer program is fed with the imaging potentially dynamicallyadjustable (including feedback from the measurement) parameters andcontrols the imaging procedure.

(5) image

Point measurement.

1D scan: Collect data from a region of interest

2D scan:

-   -   Image a 2D region of interest by scanning the foci in parallel        across the imaging plane (XY)    -   the 2D scanning starts for example at a corner of an area and is        then scanned in a raster until the area is covered according to        the imaging parameters (current implementation)    -   or it can be scanned in any other way (even random scan is        allowed), as long as the area is covered according to the        imaging parameters and the position of the foci is known by the        signal sent or received by the scanner.

A key consideration for the improvement in the measurement is that thedetector measures the sample for each scanning position, withoutoverlap.

3D scan:

-   -   move the focusing device (objective lens) in reference to the        sample along the optical axis (Z) and repeat the 2D imaging        process. Either, the focusing device (objective lens) or the        sample can be moved. Right now, the focusing device (objective        lens) is moved stepwise in regards to the fixed sample.    -   2D imaging along the optical axis (Z) can begin at any point in        the sample and end at any point of the sample within the region        of interest.        -   the current movement is though depending on the application:            -   (a) begin scanning a 2D scan of the top layer of the                tissue sample, or            -   begin scanning a little outside the sample, to be able                to determine the top, or            -   begin scanning within the sample, to prevent beam-sample                interactions (like burning or such thing), which take                place at the surface-immersion medium barrier.            -   (b) then move the focusing device (objective lens) by                means of a piezo in the direction of the sample in                increments determined by the imaging parameters            -   start at (a) again            -   Stop at a point in the sample, which is determined by                the imaging parameter            -   Move the piezo back to its original position            -   Move the sample to a different position and start                with (a) again (area imaging)    -   For some applications it is preferable, if the movement can be        reversed (starting inside of the sample and then move out), or        performed in a ‘random’ fashion, covering the whole area, as        long as the z-position is known.    -   The z position of the foci is known as the piezo position is        known    -   The 2D scanning is done while the z-scan from one position to        the next takes place or after the z-scan has completed its move        to the next position.    -   Images of 2D sections can be done alone, without any 3D movement        involved.

(6) Process images for feedback purposes (optional)

(7) display the images (optional)

(8) save the data

-   -   Process data before saving (Optionally)

(9) Post process the data (optional)

-   -   Image normalization    -   linear image deconvolution    -   nonlinear image deconvolution

(10) save the processed data (optional)

(11) display the processed data (optional)

In addition to mechanistic applications, time-resolved measurements,either alone or in conjunction with spectral measurements, can greatlyaid in distinguishing signals from different reporter probes andprocesses, such as simple scattering and non-linear scattering. Forcytometry applications, the additional information from time-resolvedmeasurements can potentially increase the number of probes which can beused simultaneously, provide images cell morphology by detection ofsecond harmonic generation, and aid in deconvolution of images fromhighly scattering samples.

FIGS. 29(a) and 29(c) relate to a deconvolution process. In FIG. 29(a)Illumination foci in the optical plane. (foci f11-f33 are illustrated inan enlarged) along with an object, illuminated by foci f22. In FIG.29(b) the detection signals are scattered along with the detection areasa11-a33. FIG. 29(c) are example of signal counts detected by theassociated channels of the multi channel detector (signal from area allis collected by the detector channel c11; a12 by c12 and so on) at acertain time point, when the focus f1 scans the center of the object(a). The relative signal distribution between the channels is dependenton the mean free path of the detection photons in the media and thepenetration depth. It is constant however, if a homogeneous scatteringdistribution is assumed (For many samples this can de assumed in thefirst approximation). Low mean free path means highly scattering, meanshigher amount of signal in other than channel c22. As the penetrationdepth into the sample increases, the chance, that a photon is scatteredon its way to the detector array increases and thus the described“optical cross talk” increases as well. More scattered light is found inthe channels, neighboring the outer channels c11, c12, c13, c21, c23,c31, c32 and c33. The signals in these detection elements are smallerthough and are not illustrated for simplicity.

FIGS. 30(a)-30(d) display a 1 dimensional (1D) deconvolutionexemplifying the final 2D deconvolution executed in the linear imagedeconvolution. For simplification only nearest neighbors are shown. Alinear convolution with a delta function with inversed side lobes (FIG.30(c) (Illustration only in along one channel number direction) resultsin a linearly de-convolved image in which only the channel c22 carries asignal. In practice, this function can either be modeled or measured. Ifthe deconvolution process is shown for simplicity only in x direction.It will be carried out in both x and y directions, and will result in animage in which only channel c22 will carry a signal.

Assuming a homogeneously scattering material (which can be assumed forsamples in the first approximation), the relative and absolute height ofthe peaks of the delta function is fixed for every channel at itsneighbors at a certain imaging depth into the sample. As a result, xyimages can be linearly de-convoluted.

The linear deconvolution of cross-talk is primarily a 2D process. Thevalues of the weighting matrix depend on several factors. The opticalcontribution to the cross talk increases with increasing penetrationdepth. Furthermore, the channels have different sensitivities, there iselectronic cross talk between channels that varies from channel tochannel and other factors influence the amount of total cross talkbetween the channels.

The cross talk for each individual channel can be determinedexperimentally. An example is where, one focus illuminates the sample ora test object and the whole array of detectors detects the signal. Atdifferent penetration depths a cross talk matrix is measured for eachchannel. This matrix is then used to carry out the deconvolution. Dataof such a measurement at the sample surface and at a penetration depthof 200 mm, can be used. The measurement is repeated for every channelfor example by moving the iris from transmitting light from one singlemicro lens to the next (in this case for channels c11 to c33). Similaralternative methods are also possible, for example by illuminating withall of the foci but using a sample with large object spacing.Furthermore, models can also replace experimental determination.

An entire 2D image consists of collections of ensembles for a non lineardeconvolution, of pixels, from each detector. The key point is thatrelationships between entire ensembles, and certain regions of pixelsbetween ensembles, can be established to constrain the variation of theweighting matrix to aid convergence without assumptions, or with minimalassumptions, of the sample or the processes which cause the variation inthe weighting matrix.

For example, continuity of the values across the boundaries of theensemble can be generally required. In the case with the minimalassumption that the objects under observation are smaller than theregion covered by an individual ensemble, the ensembles can beconsidered largely independent, except due to the cross-talk introducedby the weighting matrix. The ideal image can be recovered bysimultaneously solving for a weighting matrix which minimizes thecovariance between ensembles. In the other case where the objects underobservation are of similar size or larger than the regions covered bythe ensembles, minimal models of the object (such as from imagemorphology or segmentation of the collected image, etc . . . ) can beused to form constraints.

Additional model dependent and independent constraints can also beapplied by consideration of the planes above and below the plane underevaluation. Further constraints can also be applied to the weightingmatrix from either general (such as continuation, smoothness, sharpness,etc . . . ) or model based considerations.

While this invention has been particularly shown and described withreference to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. A multiphoton imaging system comprising: amultiphoton light source emitting light for illuminating a biologicalmaterial with multiphoton light excitation to induce a fluorescenceemission at one or more foci; a focusing lens system including anactuator coupled to an objective lens that receives light from themultiphoton light source is configured to place the one or more foci atselected depth positions within the biological material; a scanner thatprovides relative movement between one or more foci receiving theilluminating light and the material to be imaged at the selected depth,the scanner configured to place the one or more foci at selectedhorizontal and vertical positions within the biological material, theplacement of the one or more foci in horizontal, vertical, and depthpositions enabling generation of three dimensional image data; anoptical system that couples light from the scanner onto a region ofinterest of the material, the optical system including a moveable lensto adjust focal depth in the biological material; a detector system thatdetects light from the region of interest to generate image data at aplurality of different wavelengths, the detector system including aplurality of photomultiplier tube detector elements, including a firstphotomultiplier tube detector element that is optically coupled to theregion of interest with a first optical fiber and a secondphotomultiplier tube detector element that is coupled to the region ofinterest with a second optical fiber; a holder for the biologicalmaterial, the holder being mounted on a controllable stage; a sectioningsystem for sectioning the biological material; a controller connected tothe scanner, the light source, the focusing lens system, and thecontrollable stage to automatically control the imaging system; and animage processor connected to the detector system that processes imagedata detected at different focal depths, different horizontal andvertical positions, and at different wavelengths in the biologicalmaterial to generate the three dimensional image data.
 2. The system ofclaim 1 wherein the detector system comprises a plurality of detectorelements each detector element being optically coupled to the materialwith an optical fiber.
 3. The system of claim 1 wherein each detectorelement detects light from a different focal location.
 4. The system ofclaim 1 wherein each detector element comprises a photomultiplier tube.5. The system of claim 1 wherein each detector element has a detectionarea and a detector spacing separating each detector element from anadjacent detector element.
 6. The system of claim 1 wherein the detectorsystem comprises a multi-anode photomultiplier tube imaging detectorhaving at least 64 detector elements.
 7. The system of claim 1 whereinthe optical system illuminates at least 4 spatially separated foci in a2×2 scan pattern.
 8. The system of claim 1 wherein the system performsspatially multiplexed imaging.
 9. The system of claim 1, wherein theoptical system simultaneously illuminates a plurality of spatiallyseparated foci.
 10. The system of claim 1 wherein controller alternatesbetween excitation foci patterns and wherein the system illuminates fociat different depths.
 11. The system of claim 1 wherein the detectorsystem detects non-descanned light from the material in response to theilluminating light.
 12. The system of claim 1 wherein the optical systemcomprises a diffractive optical element.
 13. The system of claim 1,wherein the system illuminates foci at a first focal depth and a secondfocal depth.
 14. The system of claim 1 wherein the scanning system scansa plurality of foci in a linear array.
 15. The system of claim 1 furthercomprising a data storage device that stores image data and metadataincluding foci spacing, foci number, and scanner data.
 16. The system ofclaim 1 further comprising recording of focal positions for the scannerposition and the stage position at each focal position scanned by thescanner.
 17. The system of claim 1 wherein the system is configured toperform time multiplexed illumination, the time multiplexed illuminationcomprising illuminating a plurality of focal locations in the material.18. The system of claim 17 wherein a plurality of beam paths illuminatea plurality of foci, at least one beam path transmitting a first pulsethat is temporally separated from a second pulse, and wherein thedetector system detects the first pulse and the second pulse.
 19. Thesystem of claim 1 wherein the system is configured to perform timemultiplexed detection.
 20. The system of claim 1 wherein each opticalfiber is coupled to a separate foci of a foci pattern.
 21. The system ofclaim 1 further comprising an illumination fiber optic device thatcouples light from the light source to the material.
 22. The system ofclaim 1 wherein light from each foci is coupled to the first or secondoptical fiber with a lens element.
 23. The system of claim 1 wherein thescanner is a resonant mirror scanner.
 24. A multifocal light detectingsystem comprising: a multiphoton light source system that illuminates amaterial with a plurality of light pulses such that at least two photonsat each of a plurality of focal locations located at a selected depthwithin the material induce fluorescence at each focal location; amultifocal optical device that provides a plurality of pulsed lightbeams at a pulse rate at each of the plurality of focal locations; anoptical system including a focusing lens system having an actuatorcoupled to an objective lens that couples the plurality of pulsed lightbeams from the multifocal optical device onto a region of interest ofthe material at a plurality of focal locations at a selected adjustabledepth within the material; a scanner system configured to place theplurality of focal locations at selected horizontal and verticalpositions on the material; a control system connected to the multiphotonlight source system, the scanner system, and the focusing lens system,the control system automatically controlling the multiphoton lightsource system output power as a function of light penetration depth inthe material; a detector device that detects fluorescent light from theplurality of focal locations in the region of interest with a pluralityof detectors to generate three-dimensional image data at a plurality ofdifferent wavelengths, the detected fluorescent light being emitted inresponse to the pulsed light beam; and a processor connected to thedetector that processes the three-dimensional image data.
 25. The systemof claim 24 wherein the scanner system comprises a rotating mirror or aresonant mirror.
 26. The system of claim 24 wherein each detector in theplurality of detectors has a collection area corresponding to ascattering distribution for a corresponding focal location in theplurality of focal locations.
 27. The system of claim 24 wherein thedetector device detects time resolved data for deconvolutions.
 28. Thesystem of claim 24 wherein the processor is configured to execute acomputer program that processes time resolved data in combination withspectroscopic data to distinguish components of tissue.
 29. A method formultifocal light detection in response to light from a multiphoton lightsource system comprising: illuminating a region of interest with lightfrom a multiphoton light source system using a plurality of scanningoptical pathways to form a plurality of scanning focal locations atselected horizontal and vertical positions within the region of interestwith a multifocal optical device including a focusing lens system havingan actuator coupled to an objective lens to place the plurality ofscanning focal locations at a selected adjustable depth within theregion of interest, the scanning focal locations simultaneously movingin tissue in response to control signals from a controller;simultaneously detecting fluorescent light from each of the plurality offocal locations in the region of interest in response to pulsed lightillumination including at least two photons received at each focallocation required to induce fluorescence at a given time duringscanning, the fluorescent light being detected with a first detectorarray that detects a first spectral component having at least a firstwavelength and a second detector array that detects a second spectralcomponent having at least a second wavelength, the first spectralcomponent being spectrally distinct from the second spectral component;and processing the first spectral component and the second spectralcomponent to generate a plurality of spectral images of the tissueincluding three dimensional image data at different wavelengths.
 30. Themethod of claim 29 further comprising actuating relative movementbetween the pathways and a material by scanning with a rotating mirroror a resonant mirror.
 31. The method of claim 29 further comprisingdetecting with the first detector array having a plurality of detectorelements, each detector element having a collection area correspondingto a scattering distribution for each of the plurality of scanning focallocations and collecting time realized data and fluorescence data. 32.The method of claim 29 further comprising detecting light with the firstdetector array that comprises a plurality of photomultiplier tubedetector elements.
 33. The method of claim 29 further comprisingactuating the multiphoton light source system that emits light at awavelength such that at least two photons of light that are incident ata focal location of a material within the region of interest induce afluorescence emission from a material.